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Foreword 


When the Association of American Universities (AAU) launched its initiative to improve under- 
graduate STEM education in the fall of 2011, it set out to influence the culture of STEM depart- 
ments at AAU universities in ways that would encourage their faculty members to use teaching 
practices proven by research to be effective in engaging students and helping them learn. AAU 
aimed to address the institutional and cultural barriers impeding improvements to large intro- 
ductory STEM courses — often referred to as “weed out courses.” 


The overarching goal of the Undergraduate STEM Education Initiative was very ambitious. Trying 
to achieve large-scale systemic change at our institutions was not a typical project for the AAU. 
It was not at all clear that the association could positively affect the quality and effectiveness of 
undergraduate STEM teaching and learning at member universities. 


AAU sought an outside evaluation to assist it in understanding the association's role in improv- 
ing undergraduate STEM education at its member institutions. Such an evaluation focused on 
helping the association reflect on the Initiative’s achievements, shortcomings, and what AAU 
might do in the future to continue to make progress toward achieving the widespread improve- 
ment in the quality and effectiveness of undergraduate STEM teaching. AAU also hoped that 
an independent analysis might inform other higher education associations and scientific societ- 
ies about effective strategies for national associations seeking to help individual institutions 
achieve broad-based systemic and cultural changes. 


With support from the National Science Foundation, Dr. Adrianna Kezar, a noted expert in higher 
education change, governance, and leadership from the University of Southern California, and 
her colleagues observed and assessed the initial years of AAU’s Undergraduate STEM Education 
Initiative. This report summarizes the key findings by Dr. Kezar and her research team. 


The report's findings will be carefully reviewed by the AAU staff, the board of directors, and 
the full AAU membership to identify lessons to strengthen efforts to improve the quality and 
effectiveness of undergraduate education at AAU universities. The findings also will assist AAU 
when it embarks on future efforts in partnership with member campuses. 


The analysis demonstrates that AAU had a substantial positive effect on educational reforms on 
its campuses both for the eight project sites and for the larger network of AAU institutions. The 
prestige and status of AAU; the Framework for Systemic Change in Undergraduate STEM 
Teaching and Learning providing a conceptual model for how individual institutions might pro- 
ceed with implementing and institutionalizing educational reforms; and the Essential Questions 
and Data Sources report that helps campuses track the progress of their reform efforts were 
noteworthy. The report also identifies areas in which AAU had less effect than desired, along 
with recommendations for how to improve those areas. Finally, AAU expects that the research 
findings can assist other national higher education and disciplinary societies as they seek to fos- 
ter improvements in the educational performance of their constituencies. 
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Key lessons for national organizations for scaling improvements in 


STEM learning environments 

= Assess strengths and assets as well as limitations as an organization. 

= Use a systems approach and networks to scale change. 

= Develop a multi-theory strategy for maximum impact. 

= Understand and intentionally plan influence strategies. 

= Carefully evaluate the framing and language used to communicate the change. 

= Apply strategies to facilitate learning and adoption, and examine barriers to learning 
in change processes. 

= Consider culture change at multiple levels — the culture at the national 
organization as well as the culture at the member organizations. 

= Create distributed leadership to improve STEM education. 


For organizations engaged in scaling change, we have developed a practitioner guide with 
tools to help in this process. 


This study explored an undergraduate education initiative developed by the Association of 
American Universities (AAU) to improve the quality of undergraduate teaching and learning in 
science, technology, engineering, and mathematics (STEM) fields at its member institutions. The 
study relied on interviews, documents, and observations to examine strategies that have been 
successful in scaling and sustaining changes arising from the AAU STEM Initiative. 


The project explored two major questions: 
1. How does the AAU Initiative achieve scale in reform of undergraduate STEM teaching 
and learning? 
2. What is the role of a national association in undergraduate STEM reform and what 
approaches are perceived to have the most impact in achieving scale? 


Following are the key findings related to these questions: 


AAU used a systems approach to promote long-lasting large-scale change in STEM 
teaching. The systems approach used by AAU included three primary levels: the institutional 
level, consisting of the Initiative’s eight campus-based project sites; the AAU network level, across 
the 62 AAU member-universities as a group; and the national level, composed of various national 
agencies and groups interested in STEM teaching reform. As intended, AAU’s strategic focus on 
these primary organizational levels, as well as the various subsystems within them, helped foster 
large-scale commitment to reforming STEM education by its member institutions. 


AAU was most effective when it used strategies that leveraged the prestige of the “AAU identity,” ie., 
the most prestigious research university collective in the United States. The three most efficacious strat- 
egies were: 

= strategies aimed at utilizing the prestige and influence of AAU 

= strategies that built on the historic strengths of AAU 

= strategies that utilized a unique niche of AAU 
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When interviewees commented about missed opportunities to improve undergraduate STEM 
education, they often meant AAU not leveraging its prestige and influence, not working with 
its historic strengths (such as mobilizing presidents or issuing national reports), and not utilizing 
its unique niche (e.g. its ability to convene key stakeholders). 


These findings demonstrate that AAU appears best-positioned to impact change at the network 
and national levels, although its influence clearly was felt by the individual project sites. In par- 
ticular, interview data revealed AAU’s particular perceived strengths in working with national 
groups; helping shape national priorities and funding; helping garner media attention; and influ- 
encing administrators’ perspectives and priorities. Placing more emphasis on these may advance 
future AAU efforts. 


AAU’s most effective roles in improving STEM undergraduate teaching are based on 
leveraging its influence. The data in this report demonstrates the importance of maximizing 
AAU's organizational prestige and reputation to garner support for change at institutional and net- 
work levels. As an organization representing leading U.S. graduate-level universities, AAU can influ- 
ence the values and priorities operating across the higher education enterprise. By itself, AAU’s launch 
of the Undergraduate STEM Education Initiative helped increase the attention paid to undergradu- 
ate teaching at its member institutions and arguably at other research institutions. The strategies 
that leveraged AAU’s influence in this manner included: name dropping AAU institutions to influ- 
ence colleagues; branding the project as an AAU effort; creating competition among peer institu- 
tions; creation of a peer or comparison group (i.e., project sites and an AAU Network); media atten- 
tion; rewards and recognition; site visits; and partnering with external groups. 


AAU used framing and messaging strategies to help increase the value or at least the 
attention paid to teaching in higher education. Institutional logics are socially constructed pat- 
terns of practices, assumptions, values, and beliefs through which people attribute meaning to their 
social world. Logics govern individual and organizational action by dictating sources of legitimacy 
within an organizational field. For the last century at least, the logics governing work in research uni- 
versities emphasize the importance of research and graduate education, encouraging the faculty to 
pay more attention to research and scholarship than teaching, especially undergraduate teaching. 


Logics are often deeply embedded in organizations and the individuals within them, which can 
lead to unquestioned acceptance of norms. However, new or emergent logics can be introduced 
and legitimated through language and discourse by developing frames that resonate with their 
target audiences. For example, AAU developed language to enhance the prestige of teaching 
by, é.g., making its members “as excellent in teaching as they are in research.” This frame aligned 
with the existing values and priorities (“excellence” as a desirable standard in research universi- 
ties) of its target audience. Because of AAU’s expansive network and influence, the frame has 
been widely adopted, prompting the beginning of a shift in institutional logics. In addition, AAU 
was very successful in establishing a contradiction that helps people engage new logics. In par- 
ticular, AAU set up a major contradiction using the idea that “great institutions cannot be poor 
at teaching.” Variations of this phrase were used (research universities as the epitome of excel- 
lence for research should reflect this excellence in teaching, etc.) but all emphasized the same 
logic: As the epitomes of excellence, AAU institutions should not tolerate having a major por- 
tion of their work—undergraduate teaching—be considered poor. 
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AAU used a variety of strategies to disseminate and embed the recreated logics. These included 
the various networks that AAU had created as part of the Initiative as well as the points of con- 
tact at most member institutions as part of the AAU STEM Network. Additionally, the AAU pres- 
ident described the focus of the Initiative at each meeting. The new logic was listed on the AAU 
website, noted in listserv messages to the campus points of contact and project site leaders, 
and included in newsletters and other outreach. The Framework for Systemic Change to 
Undergraduate STEM Teaching and Learning document describing the rationale for change and 
the strategies employed made clear that AAU was an important mechanism for disseminating 
the new logic by having a concrete image and reference point for the logic in the change 
process. 


Because of the academic and research excellence of its members, AAU is one of a few actors 
in American higher education that can shape institutional logics, establish contradictions, and 
impact areas such as reward structures in the long run through changes in logics/values. For 
these reasons, AAU is an essential player in achieving systemic change in undergraduate STEM 
education. 


Rather than limiting its efforts to the original eight project sites, AAU focused from the 
start on building a network of its members to support undergraduate STEM reform. 
Research suggests that evidence of improved practices is a necessary but not sufficient condition 
to create and sustain change. Institutional and interpersonal networks, and peer interactions are 
crucial in changing ideas and practices. AAU leveraged networks to scale change at all three levels 
of the system, its campuses, subgroups on campuses, coalitions focused on STEM education, and 
external groups. These groups serve different purposes, including influence, learning, funding and 
resources, dissemination, and emotional support. This report describes promising networking prac 
tices including identifying peer networks, providing opportunities for networking, and identifying 
network subgroups. This report also demonstrates the importance of connecting multiple networks, 
providing support for networks, and developing leadership to maintain networks. The multiple net- 
work functions within the AAU Initiative would not have flourished without the planning and atten- 
tion of AAU staff. AAU was effective at creating and leveraging networks to scale change because 
networking builds on AAU’s image as representing prestigious institutions and effectively advance 
ing important policy initiatives. 


The study also identified ways that networking could be improved. The recommendations 
include: identifying the right network members; empowering the AAU STEM community to lead; 
creating network leadership by tapping existing president, provost, and dean networks; broad- 
ening on-campus teams; networking department chairs; and distributing ownership and lead- 
ership for networks. 


AAU focused on information sharing and learning in the STEM Initiative. Networks facil- 
itate information sharing and the circulation of new ideas. Although important to promote change, 
information sharing is only a precursor to learning and does not guarantee adoption of changes. 
Study data reflects that information sharing was prevalent through AAU networks, but learning 
was mentioned much less often than other strategies such as influence and framing. Key areas 
where adoption of teaching-related innovations occurred across campuses included data analytics, 
curriculum, facilities to support active teaching, pedagogy and the like. 
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AAU initially supported organizational learning by considering how to structure the project site 
work plans, including the use of data, working collectively in teams, and working across depart- 
ments. More direct AAU influences on organizational learning took place through site visits and 
AAU STEM Initiative meetings. Finally, the request for proposal (RFP) process and entities such as 
Faculty Learning Communities promoted local organizational learning. 


Several factors appear to have shaped the degree of learning and adoption of practices across AAU 
campuses. On the positive side, AAU helped establish new norms (i.e., to improve undergraduate 
STEM teaching), which had to take hold before the adoption of new practices and learning occurred. 
Potentially negative factors affecting organizational learning included perceptions that practices 
were not applicable unless invented at the specific campus (the “not invented “here” syndrome), 
competition among campuses, and the need for data from the local context to support any change. 


AAU used multiple strategies and theories of change simultaneously to help acceler- 
ate change. Among other strategies, AAU used a combination of influence, systems approaches, 
organizational learning, institutionalization, cultural approaches, and shifting norms. Multiple the- 
ories of change can help motivate and persuade campuses to change, offer them rationales to do 
so, make changes to infrastructure, and facilitate adoption of new ideas and practices. 


This study first identified the value of using several theories of action to create synergy to amplify 
the change. Second, this study also illustrated how national organizations can reflect on their orga- 
nizational identity, assets and niche in determining which theories of action to use. Third, inten- 
tional deployment of theories can assist in maximizing the effect of the theory by fully planning its 
use. In a project of this complexity, a single approach would have limited the impact, scale and 
opportunity for change. Synergy can be achieved between approaches; often one approach helped 
support another change strategy. As a respected voice in higher education, AAU was able to lever- 
age its influence (a tactic deriving from institutional theory) toward change and create new institu- 
tional logics that reshape values. Through its national prestige and influence, AAU was also able to 
leverage the open systems approach, and to impact groups such as the National Science Foundation 
and the National Academies of Science to create support for improving teaching. AAU is one of 
the few groups with the prestige to align many different players/organizations within the higher 
education system to create a powerful and aligned network of action. Using these various 
approaches together resulted in AAU’'s success in scaling change to date, but relying on any one 
strategy would have been limiting. 


AAU’s most effective role in fostering change was perceived differently by various par- 
ticipants. In developing strategies for complex change processes, it is important to consider the 
backgrounds and experiences of various stakeholders. The Framework attempted to distinguish 
stakeholders in part by identifying distinct levels of internal and external effects on STEM teaching 
as well as distinct stakeholder groups. Even so, the Framework is insufficiently detailed to provide 
specific strategies for member institutions. Instead a local needs assessment to identify stakehold- 
ers and their needs must complement the more general AAU Framework to promote effective 
change on local campuses, and to identify strategies for distinct stakeholder groups to make the 
most of their connection with AAU. 


For example, administrators were much more aware of AAU's prestige than were faculty members. 
Administrators were also more knowledgeable about the role of AAU in shaping the overall land- 
scape of higher education. As one consequence, faculty members and administrators of various 
types often differed in how they perceived AAU’s role in fostering the use of evidence-based teach- 
ing in STEM on their campuses. 
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It seems unlikely that AAU as an organization is in the best position to address various group needs 
on individual campuses. One option to meet the needs of multiple constituent groups is to con- 
duct a needs assessment and determine which needs are best met by AAU and which are best met 
by other actors. AAU could consider partnerships or brokering with other organizations that might 
be better able to meet these needs. 


AAU’s work was similar to, but also departed from, traditional theories of university reform. 
There are three key aspects to traditional theories of scaling reform examined in this study: de/ib- 
eration, networks, and external supports. Opportunities for deliberation and dialogue among 
professionals — such as convenings, conferences, and meetings — develop motivation, interest, 
and ownership. Through dialogue, norms and values are explored and changed as people accept 
new ways of doing things. Networks connect people to others with similar ideas and also pro- 
vide change agents with the information, incentives, and social capital that facilitates the change 
process. Networks also reduce risk-taking by having groups of people experiment together. 
External mechanisms of support can take many forms ranging from concept papers and frame- 
works to funding, awards, and recognition necessary to help sustain change agents in the face 
of entropy and even negative dynamics. These strategies acknowledge that individuals (or even 
individual campuses) cannot successfully act alone to scale changes and that their environments 
must be reshaped to support the changes themselves. An individual campus is unlikely to act 
outside the norms, messages and incentives developed within the broader enterprise of higher 
education. Disciplinary societies, national organizations, and regional groups all shape and influ- 
ence campus behaviors, including individual faculty behaviors. Aligning messages among groups 
provides motivation for change. 


To some extent, AAU utilized all three components of theories of scale — deliberation, networks, 
and external supports. AAU created a network of networks that scaled changes across many 
departments, campuses, and organizations. The AAU STEM Initiative also provided important 
external supports. AAU was not as successful in creating deliberation as it was with the other 
two strategies. AAU’s most effective strategy for scaling reform involved its status as a prestige 
organization with the power to reshape the higher education enterprise — a significant departure 
from traditional theories of scaling reform. 


AAU can maximize its impact on STEM education reform by finding ways to incorporate the 
quality of teaching into its overall mission. Data suggest that AAU can be a stronger force for 
improving the quality of instruction in higher education by leading the charge. The AAU STEM 
Initiative is certainly one part of that message, as is making clear that AAU is committed to main- 
taining its effort in improving education for the long run. More complex are steps to add a teach- 
ing component to membership criteria. Interviewees tended to support a modification in AAU 
membership. Many member institutions look to AAU for clues about institutional performance. 
Finding additional ways to make clear that undergraduate teaching quality and effectiveness is a 
fundamental expectation for AAU members is an important step in this process. 


Lessons for National Organizations 

Although the study provides important feedback for AAU to hone its strategy, larger lessons 
about the role of national organizations in reforming undergraduate education also emerged. 
The concepts of mapping strengths and capabilities, having concrete plans about influence, opti- 
mizing networks, and practices for maximizing learning are likely transferable to these other set- 
tings. Furthermore, many of the principles that emerged could also be utilized in regional or 
local organizations. The report ends with key recommendations for other external organizations 
and stakeholders interested in scaling changes. A practitioner's guide is available online. 
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Section | 
Background, Literature 
and Methodology 


From 2014-2017 The Pullias Center at the University of Southern California, 
generously funded by the National Science Foundation (NSF), studied the 
roles and influences of the Association of American Universities (AAU) in 
scaling teaching innovations in introductory science, technology, engineer- 
ing, and mathematics (STEM) courses at its member institutions. 


AAU's principal mechanism in this effort was its Undergraduate STEM Education Initiative (AAU 
STEM Initiative or Initiative) funded by The Leona M. and Harry B. Helmsley Charitable Trust, the 
NSF, Northrop Grumman Foundation, the Burroughs Wellcome Fund, Elsevier, and Research 
Corporation for Science Advancement. 


Before presenting findings, we review the AAU STEM Initiative, the national context in which it 
operates, and how it differs from past efforts to improve undergraduate STEM education. Next 
we discuss the relevant literature and its use in the study design and interpretation of results. 
We present the findings of the study in Section Il. 
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Background and Overview of Study 


The need for teaching improvement in undergraduate STEM disciplines and potential ways to 
redress instructional inadequacies have received increased attention and taken on new urgency 
in recent years (see e.g., President's Council of Advisors on Science and Technology). We have 
begun to see a shift to a much more coordinated vision about needed reforms and how to 
achieve them (see e.g., American Association for Advancement of Science, 2011; Howard Hughes 
Medical Institute, 2009; National Academies, 2010; National Science Foundation, 2010). This 
shift has been driven in part by new scholarship demonstrating the effectiveness of evi- 
dence-based active learning techniques in comparison to the sage on the stage, or expert lec 
turer who transmits knowledge (National Research Council, 2012) as well as widely disseminated 
recommendations for improving STEM instructional practices (President's Council of Advisors 
on Science and Technology). 


Despite abundant demonstrations of effectiveness, student-centered, evidence-based teaching 
practices have not been widely adopted in introductory undergraduate STEM courses. The rea- 
sons are complex, including deep-seated customs, faculty rewards structures, the competitive 
pressures to stay productive in research, as well as the need to prepare and persuade faculty to 
use these new approaches and for students to engage with them. Most efforts to engage fac 
ulty in using evidence-based active learning teaching practices have focused on attempting to 
change individual faculty members’ instructional behaviors in isolation. Much less attention has 
been paid to organizational culture and faculty rewards. Fairweather (2008), in his report to the 
National Academies Research Council Board of Science Education, noted that teaching reforms 
focused solely on individual instructors rarely lead to widespread reform. Instead, “the instruc 
tional role [cannot] be addressed independently from other aspects of the faculty position, par- 
ticularly research, and from the larger institutional context” (p.3, 2008). Fairweather (2008) and 
Austin (2011) recommend that reforming 


STEM instructional practice requires engaging The overall objective of AAU’s 


institutional leaders such as department F 
chairs, deans, and presidents in rethinking Undergraduate STEM Education 


institutional structures and culture. Austin Initiative is to influence the culture 


(2011) also suggests that external groups such of STEM departments at AAU 
as disciplinary societies, national associations, 


government agencies, and employers willbe | UNiversities to encourage and 
necessary to help support and leverage support the faculty to use teaching 
change within institutions. ‘ 

practices proven by research 
In 2011, AAU launched a five-year initiativein tobe more effective in engaging 


partnership with member institutions to - F 
improve undergraduate STEM teaching and students in STEM education 


learning. In contrast to traditional reforms that and in helping students learn. 
focused primarily on reforming the teaching 

of individual faculty members, the AAU Initiative focused on reforming the primary academic 
unit in the institution, the department, and incorporating both curricular and cultural elements 
in the improvement of STEM undergraduate education. The overall objective of AAU’s 
Undergraduate STEM Education Initiative’ is to influence the culture of STEM departments at 


ai 


AAU published a Progress Toward Achieving Systemic Change, a five-year status report on the AAU Undergraduate STEM 
Education Initiative in October 2017. The report documents the results of the AAU STEM Initiative to date. AAU is committed 
to expend the initial effort indefinitely by integrating continued support for undergraduate STEM education improvement into 
its ongoing portfolio of work. 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 7 


AAU universities to encourage and support the faculty to use teaching practices proven by 
research to be more effective in engaging students in STEM education and in helping students 
learn. The goals of AAU’s STEM Initiative are to: 


1. develop an effective analytical framework for assessing and improving the quality of STEM 
teaching and learning; 

2. support project sites at a subset of AAU-member institutions to implement the framework; 

3. explore mechanisms that institutions and departments can use to train, recognize, and 
reward faculty members who want to improve the quality of their STEM teaching; 

4. work with federal research agencies to develop mechanisms for recognizing, rewarding, 
and promoting efforts to improve undergraduate learning; and 

5. develop effective means for sharing information about promising and effective undergrad- 
uate STEM education programs, approaches, methods, and pedagogies. 


In collaboration with its member universities, AAU developed a Framework for Systemic Change 
in Undergraduate STEM Teaching and Learning (Framework) to guide institutions in their com- 
mitment to facilitate change in undergraduate STEM education. This required identifying key 
levels of change, the agents of change, and the mechanisms of change, as well as the models 
for scaling and sustaining it, drawing from literature on institutionalization. The core of AAU's 
Framework is pedagogy: the practices used by faculty members to engage students and guide 
and support their learning. To successfully enact and sustain the use of evidence-based teaching 
techniques, two additional elements are necessary; scaffolding, or support, for both faculty and 
students; and larger cultural change to facilitate changing teaching practices. Ultimately, the 
Framework is intended help the various individuals and organizations (faculty members, depart- 
ment chairs, college and university administrators, and institutions) who work together to improve 
undergraduate STEM teaching and learning. See Figure 1 — The Framework: 


Figure 1 
The Framework 


CULTURAL CHANGE 


SCAFFOLDING 


PEDAGOGY 
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From among 31 AAU universities that submitted concept papers, AAU selected eight member 
campuses to serve as project sites*. Several criteria were considered when choosing sites, such 
as the degree of departmental and faculty engagement, institutional commitment, likelihood of 
sustained organizational change, and commitment to evaluation and assessment. As a collec 
tive, the eight project sites represented the diversity of the AAU membership (e.g., public and 
private universities, large and small enrollments) and proposed projects that addressed the full 
range of elements outlined in the Framework. Over a three-year period, AAU STEM Project Sites 
served as laboratories to implement STEM education reforms. Project sites are the first phase of 
a larger AAU commitment to encourage broad-based reform of undergraduate teaching prac 
tices at AAU universities and beyond. 


A crucial step in disseminating STEM educational reforms beyond project sites was AAU’s forma- 
tion of an AAU STEM Network. AAU envisioned a collaborative network to link member institu- 
tions grappling with similar challenges in improving STEM teaching and learning for undergrad- 
uate students. The network met annually during the initial five years of the AAU STEM Initiative 
with plans to continue meeting into the future. Research by Fairweather (2008), Eckel and Kezar 
(2003), and Kezar (2001) endorses the importance of relationships and networks in changing fac 
ulty and administrative behavior especially when the collaboration is among peers as well as high 
prestige individuals and organizations. Austin (2011) and Kezar (2001) in particular identify net- 
works as an important alternative to (often ineffective) strategies based solely on empirical evi- 
dence as a motivator for changing faculty behavior. Social relationships and networks are among 
the strongest motivators for changing ideas and practices (Valente, 1995; Rogers, 2003). 


Complementing these implementation efforts is AAU’s work on metrics. With support from a 
two-year grant from the NSF “Widening Implementation & Demonstration of Evidence-Based 
Reforms” (WIDER) program, AAU developed a set of baseline measures that project sites, and 
other institutions, could use to document the current status of undergraduate teaching and learn- 
ing and monitor the effects of reforms (see Essential Questions and Data Sources released in 2017). 


At the national level AAU works closely with other key organizations involved with undergrad- 
uate STEM reform including the Association for the Advancement of Science (AAAS), Association 


of American Colleges and Universities — : 
(AAC&U) Project Kaleidoscope (PKAL), and Finding ways to align norms 


the Association of Public and Land-grant geross peer institutions, and 


Universities (APLU). By attempting to align eerie . ene. sewer 
important national norm-driving groups, AAU linking with disciplinary societies 


learned from the success, for example, of the and other key national groups 


Accreditation Board for Engineering and is important to reinforce norms 
Technology (ABET) in encouraging more 


attention on undergraduate programs by that value improved STEM 


shifting accreditation criteria to focus on stu- undergraduate education. 
dent learning outcomes (Lattuca et al., 2006). 


In addition to aligning norm-driving groups within an institution—programs, departments, and 
colleges or schools—and finding ways to align norms across peer institutions, linking with dis- 
ciplinary societies and other key national groups is important to reinforce norms that value 
improved STEM undergraduate education. 


2 Brown University; Michigan State University; The University of Arizona; University of California, Davis; University of Colorado 
Boulder; University of North Carolina at Chapel Hill; University of Pennsylvania; and Washington University in St. Louis 
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A crucial cultural barrier to STEM reform within research universities is faculty members’ per- 
ceptions of the relative importance of teaching. This challenge is especially true for Introductory 
STEM courses which often enroll students outside the major; i.e., the relevant norm here is not 
only the value of teaching but the value of teaching introductory students outside of one’s major. 
Fairweather notes “that resistance to adopting more effective teaching strategies in large part 
derives from the perception of STEM faculty that the teaching process is at odds with the 
research process, and that research is more interesting and more valued at their institutions” (p. 
5, 2008). As Austin (2011) identified, norms for faculty behavior are shaped at multiple levels — 
department, institution, discipline and through other key external groups such as the AAU. By 
adopting its Framework, the AAU Initiative attempts to incentivize its member institutions to 
address the ways that faculty norms about teaching are developed and reinforced by various 
agencies within and outside the institution. 


AAU is uniquely positioned, both within the context of its own initiative and in the wider higher 
education community, to reshape faculty norms and work with other external agencies and dis- 
ciplinary societies to align those norms. Furthermore, AAU can achieve disproportionately sig- 
nificant influence on U.S. academic institutions in part because of the long-term prestige and 
status market in which academic institutions operate (Fairweather, 1996). In reviewing the his- 
tory and sociological trends of higher education, Riesman (2001) famously quipped that the top 
80-100 institutions in the country have consistently shaped the norms and behaviors of all other 
colleges and universities in the previous 200 years. In this context, a commitment by prestigious 
research universities to improve undergraduate STEM education provides a highly influential role 
model example for other institutions to follow. 


This research project sought to use the AAU STEM Initiative as a real-time, field-based innova- 
tion to examine the role that AAU can play in achieving scale of evidence-based teaching prac 
tices in undergraduate education. The main objective is to understand how the AAU STEM 
Initiative achieves scale of improving undergraduate STEM teaching and learning. The research 
questions are: 


=" How does the AAU Initiative achieve scale in reform of undergraduate STEM teaching and 
learning? 

= What is the role of a national association in undergraduate STEM reform and what approaches 
are perceived to have the greatest impact to achieve scale? 


In the next section the literature used to frame the study is described, followed by our methodology. 
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Literature Review 


This study was framed initially by literature on scaling up innovations in higher education (Kezar, 
2001; Kezar, 2011). The study was both deductive and inductive. As we conducted our initial review 
of the documents and began observing the Initiative, we examined additional theories that might 
help explain observed phenomena including institutional theory, organizational learning, and sys- 
tems theory. We iteratively tested concepts until we found those most clearly reflected in the data. 


Since very little is known empirically about bringing innovations to scale within higher education, 
this study of the AAU Initiative can potentially add considerably to the literature. Most studies have 
focused on individual or institutional change rather than on macro changes scaled across an insti- 
tution or institutions (Kezar, 2011). Most research about scaling-up innovations (programs/inter- 
ventions) comes from policy research and applied subfields such as international or community 
development (Dede, 2006). This study draws on knowledge about scaling reforms from these 
fields as well as the limited literature in K-12 


schools, which is the closest literature baseon Since very little is known empirically 


educational institutions available (Adelman & maid bilgi ; ti t | 
Taylor, 1997; Dede, 2006). about bringing innovations to scale 


Hetining Scale. Scaied eee within higher education, this study of 
efining Scale. Scale (sometimes referred to code ne , 
as scale-up) typically has been defined as the AAU Initiative can potentially add 


broader reach of a practice being used by more Considerably to the literature. 
individuals over time (Elmore, 1996). Effective 

scaling involves depth, sustainability, soread., and shift in the ownership of the reform (Coburn, 
2003). Depth is achieved when innovations reach consequential change in classroom and school 
practice, impacting the beliefs of teachers and underlying assumptions about the nature of teach- 
ing and learning. Spread refers to the dissemination of the innovation in its complex forms (e.g., 
affecting teacher consciousness and group interactions). Shift in reform ownership is evident when 
the innovation moves from an external push for reform to its acceptance and ownership by local 
educators and their institutional leaders. Depth, spread, and shift in reform ownership combine to 
lead to sustainability. 


Characteristics of Scale. The AAU STEM Initiative is consistent with research associated with scal- 
ing change in education, including a focus on campus systems such as faculty rewards and work pri- 
orities, promoting new logics or values about teaching, and developing a network of peers to rein- 
force campus reforms (McLaughlin & Mitra, 2002). Two primary characteristics of widespread use or 
scale of an innovation are conditions that motivate and sustain. Crucial components of motivation 
are incentives, ownership, and being context based/organically created (Healey and Destafano, 1997; 
McLaughlin & Mitra, 2002; Samoff et al., 2003). For professional employees such as faculty mem- 
bers incentives can range from seed funding, reward structures, merit pay, and course releases (Kezar, 
2001). For the AAU Initiative how universities deal with the incentives in favor of research and schol- 
arship in reward structures is crucial to effective scaling of reforms in undergraduate STEM educa- 
tion (Fairweather, 2008; Kezar, 2001). Internal ownership of reforms (in contrast to viewing the reform 
as externally imposed) is another element of successful scaling of reforms (Coburn, 2003; Datnow, 
et al, 1998; Elmore, 1996; McLaughlin & Mitra, 20021; Palmer, 1992; Samoff, Sebante & Dembele, 
2003). Samoff, Sebante, and Dembele (2003) noted that reforms are inherently context- based, par- 
ticularly in education. Innovations in one setting are not easily transferred to another setting unless 
they can be modified or adapted to that setting (Datnow, et al., 1998). Scaling of a reform is most 
effective when the reform is developed organically within a school or setting rather than imposed 
from the outside (Datnow, et al., 1998; Healy, & DeStefano, 1997; Lee, & Luykx, 2005). 
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Two key conditions are important for sustainability: implementation context and norms (Coburn, 
2003; Elmore, 1996; McLaughlin & Mitra, 2002). Ignoring the implementation context is a bar- 
rier to scaling of reforms (Datnow, et al., 1998; Dede, 2006; Lee, & Luykx, 2005). For example, 
Elmore’s (1996) evaluation of 20 years of NSF-funded school reforms found that school incen- 
tive structures worked against scaling change in core activities. Relevant examples for the AAU 
Initiative include misaligned reward structures, lack of incentives, and curricular policies. Reform 
strategies that do not take into account local norms often fail to scale (Coburn, 2003; Elmore, 
1996; Samoff, Sebante & Dembele, 2003). Good practices should be openly and publicly debated 
more regularly — becoming part of the value system (Dede, 2006; Elmore, 1996). 


In this study, we examine the ways that the AAU STEM Initiative is perceived as successfully 
addressing motivation and sustainability as well as the subconstructs of incentives, ownership, 
context based/organically created, implementation context and norms. In this context overcom- 
ing resistance is as important as creating incentives for reforms (McLaughlin & Mitra, 2002). 


Strategies for Achieving Scale. Kezar’s (2011) review of research on scaling educational reforms 
identified three key levers for success: deliberation and discussion, networks, and external support 
and incentives. These strategies help overcome barriers to scaling reforms. Table 1 demonstrates 
the relationship between the strategy and the successful characteristics of scale up. 


Table 1: How each strategy addresses the characteristics of successful scale-up 


Areas of successful 


Deliberation Networks External support 
scale up 
Motivation X X 
Sustainability X X 


Deliberation and discussion: One strategy is creating opportunities for deliberation and dia- 
logue among peers — convenings, conferences, seminars, and meetings. Faculty members are 
more likely to adopt a new pedagogy when they engage in a process to help them understand 
the necessity of the change (Elmore, 1996; Kezar, 2011). Norms and values can be influenced 
by dialogue with peer professionals, which helps participants make sense of the change (Senge, 
1990). Deliberation can increase commitment to reforms by engaging participants more deeply 
in the learning process about the reforms (Kezar, 2011). Deliberation helps scaling because it 
allows for adaptation that accounts for cultural and structural differences within institutions 
(Samoff, Sebante & Dembele, 2003). 


Networks: Networks connect people to others with similar ideas as well as providing change 
agents with information to help them promote the change process locally (Datnow, et al., 
1998; Dede, 2006; Lee, & Luykx, 2005). Research has shown that people interacting with 
others in networks become more open to adopting new approaches to solve problems (Tenasi 
& Chesmore, 2003). Networks also can provide external motivation for innovation when local 
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incentives are weak (Kezar, 2011). By connecting participants with other innovators, networks 
help overcome the challenges of scale confronted by isolated individuals or organizations 
(Rogers, 2003). Networks help participants feel part of a collective strategy rather than being 
an isolated innovator (Rogers, 2003; Valente, 1995). Networks also can provide information 
about using social capital to overcome barriers to reform (Burt, 2000). Social capital is the set 
of resources (e.g., knowledge, influence that people possess, finances) embedded within 
social relations/structures which can be mobilized by an actor to increase the likelihood of 
success in purposive action (Daly & Finnigan, 2009). 


External supports: External organizations can provide individual campuses resources and 
other forms of support (e.g., concept papers, frameworks, funding, awards and recognition) 
to encourage change in the face of entropy or negative dynamics, and can also enhance the 
sustainability of change (Datnow, et al., 1998; Dede, 2006, Kezar, 2011). Individuals (or even 
individual campuses) often cannot successfully act alone to scale changes. External environ- 
ments play key roles in supporting the reforms because an individual campus is unlikely to 
act outside the norms and incentives held by peer institutions and the broader enterprise of 
higher education. Disciplinary societies, national organizations, and regional groups (e.g., 
accreditation agencies) all influence the behaviors of administrators and instructional faculty 
members (Austin, 2011; Kezar, 2001). These supports increase the likelinood of scaling of 
instructional reforms. 


Distributive rather than top down forms of /eadership are the most effective in scale-up strat- 
egies (Palmer, 1992; Senge, 1990). Distributed leadership is enacted by groups of people 
rather than depending solely on a particular position or authority. It draws on leadership 
across and sometimes beyond the institution. Distributed leadership includes faculty, profes- 
sional association employees, staff at AAU, and those in formal positions of authority such 
as administrators or policymakers. 


In addition to the theories of scaling discussed above, we found additional theoretical per- 
spectives useful in analyzing the effects of the AAU Initiative — institutional theory, organi- 
zational learning, systems theory, and institutionalization. 


Institutional Theory. Institutional theory, which fit much of our emerging data, describes why 
change occurs as a complex interplay of strong normative internal cultural dynamics with vary- 
ing external logics that push for new ways of doing things. Institutional theory suggests that 
individual change agents and institutional leaders are strongly influenced by internal forces of 
inertia, but when external norms change — in this case from peer institutions and relevant aca- 
demic disciplines — this impacts internal cultural dynamics (Powell & DiMaggio, 1991). Institutional 
theory also suggests that colleges and universities — as institutions with long-standing missions 
tied to societal goals — will change more slowly and less often than other types of organizations. 
Our research demonstrates that AAU leveraged external logics related to valuing teaching at 
research universities to helo promote institutional change. 


In addition to the nation-state, market factors, and other societal factors, various actors within 
the higher education universe also affect institutional decision-making (Powell & DiMaggio 
1991). These actors include accreditation, disciplinary societies, and policy groups such as the 
State Higher Education Executive Officers. Especially important for AAU institutions is the pres- 
tige market in which U.S. higher education operates. This prestige market places greater value 
on faculty research productivity and to a somewhat lesser degree on student selectivity. These 
values are reinforced by disciplinary societies and by faculty rewards within academic 
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institutions (Boyce 2003). Institutional theory assisted us in developing a comprehensive per- 
spective of change that integrates important external factors with key internal dynamics to 
explain the impact of the AAU Initiative on its membership. It also demonstrates how practices 
become embedded, or institutionalized, in the structures and processes of organizations, which 
makes it difficult to change values and systems once they become embedded. 


As an important external actor, AAU’s strategy in the STEM Initiative was to align the various 
internal and external players to emphasize and reinforce excellence in undergraduate teaching 
as a highly visible and desirable goal for its member institutions. Beyond its membership, AAU 
also hoped to leverage the isomorphic power 

of research universities to shift and make the AAU’s strategy in the STEM 


norms within higher education more balanced | njtiative was to align the various 
between teaching and research, and to insti- 


tutionalize the notion that faculty should be internal and external players to 
as excellent in teaching as they are in research. emphasize and reinforce excellence 
Organizational Learning. Organizational ah undergraduate teaching — 
learning is the study of whether, how, and under highly visible and desirable goal 
what conditions organizations can be saidto for its member institutions. 

have learned. Organizations are seen to change 

when individuals learn and collectively reorient the way that they approach conducting work. From 
the beginning of the Initiative the AAU leadership viewed the 8 project sites as laboratories to try 
out new practices. AAU sought to expand sharing of these reforms across the project sites and more 
broadly across AAU member institutions by forming the AAU Network. Four of the five main objec 
tives of the AAU STEM Initiative relate in some way to organizational learning, including developing 
meetings for information sharing about best practices in STEM reform, working with institutions and 
departments to train their faculty in evidence-based teaching, and supporting project sites to imple- 
ment the framework (which also could be seen as a learning tool). Data collection and sharing the 
information across member institutions was also part of encouraging organizational learning. A major 
goal of the project was to develop measures to help institutions assess the use of evidence-based 
instruction and monitor the effects of reforms on these instructional practices. 


Systems, Open Systems and Institutionalization Approaches to Scale. The AAU also 
articulated a systems approach to change. Systems theories examine the inter-relationship of 
various subsystems within an organization and how organizations are interconnected. From this 
perspective, professional development alone is insufficient to encourage evidence-based instruc 
tion because classroom practices are also tied to incentive systems, departmental norms, facil- 
ities, campus priorities, and student expectations. 


AAU also incorporated an open-system theory of change that connected internal educational 
reforms with various external actors. Although several studies of change in higher education con- 
sider the effects of internal systems like reward structures (Kezar, 2013), few studies incorporate 
open systems theory, especially in studies of teaching and learning (Kezar, 2013; Zemsky, 2013). 


The systems approach is present in the five objectives for the AAU STEM Initiative that operate 
at multiple levels — project demonstration sites, the AAU STEM Network, work with federal 
agencies, and the broader higher education system. The most well-articulated level within the 
systems approach is the institutional system, where AAU’s Framework required project sites to 
identify key levels of change, the agents of change, the mechanisms of change, and models for 
scaling and sustaining change. 
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Additionally, the project was informed by theories of institutionalization, which examine the 
ways that institutional structure and culture shape and frame implicit or tacit activities. Change 
requires reshaping the underlying structures and culture. Institutionalization theory suggests 
that the alignment of leadership and institutional infrastructure — including tenure and promo- 
tion requirements, professional development and classroom assessment — with educational 
reforms are required to sustain the reforms. 
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Methods 


Overall Methodology. Consistent with previous research on scaling innovations (e.g., Coburn, 
2003; Elmore, 1996; Samoff, Sebante & Dembele, 2003), this study used a qualitative design 
based on interviews and observations to examine perceptions of motivation, sustainability, and 
strategies meant to achieve scale. Studies of scaling innovations are best done during the inno- 
vation while memories and events are fresh in the minds of study participants (Dede, 2006; 
McLaughlin & Mitra, 2002). 


Document Analysis. The first year of the research project focused on document analysis (while 
simultaneously conducting observations) to better understand the Initiative — how it was framed 
and communicated, what strategies were used, how implementation was occurring, and the like. 
We reviewed many documents from several years of the AAU STEM Initiative including annual 
reports; correspondence about the Initiative; key documents such as the project framework, site 
visit notes, survey data about teaching practices; webinars captured through video hosted by AAU; 
and multiple meeting notes from AAU STEM Network, AAU STEM Project Site, Advisory Committee, 
and AAU project team meetings. In total, we reviewed more than 10,000 pages of documents to 
situate more comprehensive analyses of subsequent interview and observational data. We contin- 
ued to review documents developed in the final two years of this study. 


Observations. The study began with 1 % years of observation across the AAU Initiative by 
project research staff prior to interviewing participants in the Initiative. The observations were 
used to validate and extend interview data and documentation. Observations also provided con- 
text information to help interpret interview data. Additional observations were conducted for 
one year concurrent with the interview data. 


Extensive field notes were taken at the following events: 

= Bimonthly meetings by the AAU STEM Initiative project team over 2 2 years — 
28 meetings in total 

= Annual workshops for the AAU STEM Project Site leadership teams 

= Annual AAU STEM Network conference 

= Annual in-person and conference call meetings of the AAU STEM Advisory Committee 

= Larger convenings focusing on special topics or themes such as the development of 
metrics to evaluate teaching 

= Site visits to the eight AAU STEM Project Sites during year three (only a select number 
were conducted) 


= National workshops/meetings when the AAU STEM Initiative was involved 


Observations focused on the strategies used (particularly networking, convenings, and leadership) 
to scale-up educational reforms achieved through the three key principles noted in the literature: 
introduction of new ideas, social capital, and risk-taking. Through observation, we documented 
instances of new ideas circulated at meetings, the ways that solutions to problems were shared, 
and the ways that individuals expressed willingness to confront the status quo on their campus. 
These observational data were crucial to validate claims made by respondents during interviews as 
well as documentation. Observation of the convenings and meetings also shed light on whether 
discussion and deliberation (@.g., network meetings, local faculty learning communities) can help 
change norms and cultures on campuses. Observations were also uniquely situated to help us judge 
how networking among campus leaders (central administration, deans, department chairs) enabled 
them (or did not enable them) to interact with colleagues, align messages, describe values, and 
think about incentives. 
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Interviews and Sample. We interviewed four groups of individuals to understand the charac 
teristics (e.g. address underlying norms) and strategies (¢.g., convenings, relationship/network) that 
best support scale: (1) AAU STEM Initiative leaders (AAU staff responsible for the Initiative and the 
Advisory Committee for the Initiative); (2) selected faculty and administrators from the STEM 
Initiative project sites; (3) a comparable sample of faculty and administrators from non-project sites 
who are part of the AAU STEM Network; and (4) collaborators from outside organizations assist- 
ing AAU in the Initiative. In total, the study included 104 interviews (48 from AAU STEM Project 
Site teams, 26 AAU STEM Network members from non-project sites, 13 key AAU staff, and 17 
individuals from collaborating organizations). We discuss the rationale for this sample below. 


We interviewed al/ key personne! directing the Initiative, including the two co-project leaders 
of the AAU STEM Initiative, three AAU staff members, the planning team, and advisory com- 
mittee members who were integral to advancing the AAU STEM Initiative. Key personnel 
described their observations about creating and implementing the Initiative as well as their per- 
ceptions about what seemed to be leading to scale. 


For each of the eight AAU STEM Project Site campuses we interviewed between 6 and 10 indi- 
viduals. These interviews included the co-project site leadership, at least one department chair 
of a participating department, and two or three faculty members engaged in reforming an intro- 
ductory STEM course. The project site leadership were best situated and knowledgeable about 
project activities and about tracking implementation and scale-up. The project site leaders also 
were in a position to judge the effects of external forces (including the AAU) on local project 
activities. To obtain additional perspective, we also spoke with two or three faculty members on 
each campus who have been involved with the Initiative but not with the planning team. 


The AAU STEM Network included all 62 AAU institutions. Each AAU campus has a point of con- 
tact who is regularly informed about the AAU STEM Initiative and invited to participate in the 
AAU STEM Network convenings. We interviewed 26 points of contact, some who regularly 
engaged in the AAU STEM Network activities and some who were less active. These individu- 
als spoke directly to the influence of those events on their ideas for pedagogical innovation, 
support for pedagogical innovation, and leadership strategies for creating broader scale imple- 
mentation on their campuses. These interviews allowed the research team to better understand 
the mechanisms of scale affected by the AAU STEM Network. 


Lastly, we identified and interviewed collaborators. These individuals came from other national 
associations, disciplinary societies, and federal agencies engaged in national efforts to reform 
undergraduate STEM education. Collaborators provided an outside perspective on how the AAU 
Initiative has (or has not) reshaped norms about undergraduate teaching in colleges and 
universities. 


Interview protocols were customized for each group, but each protocol contained shared or core 
themes. These themes took into account the key constructs reviewed in the literature, includ- 
ing participants’ involvement with the Initiative, challenges and facilitators of scale, evidence of 
scale, perceptions of the Initiative, what has worked and not worked, and what is in the pro- 
cess of being sustained. Unique questions for each group included: (1) AAU key personnel: What 
was the history and development of the Initiative. (2) Collaborators, Network members, and 
Points of Contact: Awareness about the Initiative and its details. Prior to field testing the draft 
interview protocol was reviewed by a Project Advisory Board consisting of relevant higher edu- 
cation scholars and practitioners. 
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Interviews were approximately one hour in length and conducted via telephone. Some interviews 
with project teams took up to two hours. All interviews were recorded and transcribed. 


Data Analysis. The qualitative data (both interviews and observation) was analyzed using 
HyperResearch; a qualitative software program that helps manage and analyze large amounts 
of qualitative data. We used the Kincheloe and Berry (2004) process of bricolage to analyze 
qualitative data. This approach involves utilizing multiple theoretical perspectives to best under- 
stand a complex phenomenon. It avoids reductionism by not taking single theoretical or narrow 
approaches to interpretation. Instead, various theories are overlaid to interpret data both sepa- 
rately and then simultaneously. This approach acknowledges the complexities of social life and 
the need to bring in many different interpretive frameworks to best understand complex social 
processes. Bricolage is a useful approach to analyzing complex processes such as educational 
reforms that do not fit neatly into a single psychological, sociological, or organizational disci- 
plinary box. We overlaid ideas from institutional theory, systems theory, theories of scaling 
change, organizational learning, institutionalization, and cultural theories, all addressing change 
from different vantage points, to explore the Initiative from various perspectives. 


We employed the Boyatzis (1998) deductive and inductive thematic coding strategy. Inductive 
themes were developed by reviewing the data in the first two years before formal coding started 
in the third year of the research study. Institutional theory, for example, was added to our cod- 
ing scheme early on because its importance was evident during observation of events. We began 
with a list of codes informed from the initial literature review on scaling changes in higher edu- 
cation (e.g., incentives, networks, deliberation). We then added literature on institutional the- 
ory and organizational learning as it emerged in our informal analysis. Our final code list used 
for analyses was developed over a two-year period and included 98 codes drawn from both the 
literature and from data collection. 


Trustworthiness and Limitations. This study design draws on systematic approaches to 
studying scale proven valid in other research. It emphasizes interviews and observations to help 
to explain the underlying mechanisms related to motivation, ownership, norms, and sustainabil- 
ity (Dede, 2006). We used several methods to address trustworthiness. First, observations were 
carried out for 1 % years prior to conducting interviews to ensure that the researchers were 
familiar with the Initiative and the contexts in which it took place. This step is an essential com- 
ponent of interview validity. Second, observation and interview data were compared to provide 
multiple data points for assessing results (triangulation). Third, coding reliability was addressed 
by having several researchers code the same data and making sure that the results agreed. 
Fourth, we received feedback from our advisory board on the design, observation and interview 
protocols, sample, and data interpretations. The study was limited by little exposure to project 
sites for impromptu conversations with campus participants and a sample for each project site 
skewed toward project participants. 
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Introduction We begin the findings section (Chapters 4 and 5) with a review 
of AAU’s systems approach to facilitate scaling change. We examine three 
primary organizational levels: 1) institutional or campus-based level, 2) AAU 
STEM Network level, and 3) macro level including national agencies with a 
stake in improving undergraduate STEM education. We also examined sub- 
systems with each level such as departments within an institution. 


Chapter 6 highlights one of our main findings — that AAU’s role in scaling improved teaching 
can best be supported through influence strategies. In this section, we review data that sup- 
ports the importance of maximizing AAU’s prestige to motivate for change. Following influence, 
Chapter 7 highlights AAU’s work to frame and “message” the work of improving undergradu- 
ate teaching. Framing the work and developing a compelling rationale for the change were 
important processes for changing underlying values about the importance of undergraduate 
teaching at AAU institutions. 


Chapter 8 reviews AAU’s role in leveraging networks to scale reforms. The data strongly sug- 
gested that AAU’s most effective strategies for reforming STEM education focused on systems 
change, influence, networks, and framing/logics. Chapter 9 describes findings related to the 
final major factor for scaling reforms — learning — including challenges that emerged for AAU 
while acting as a learning facilitator. Chapter 10 examines how AAU used multiple approaches/ 
strategies to scaling change simultaneously and some of the benefits to doing so. These bene- 
fits included helping motivate and persuade campuses to change, offering them a rationale to 
do so, encouraging improvements in infrastructure, and helping campuses facilitate adoption 
of teaching reforms. We review synergies that can be created through multiple theories as well 
as some conflicts that can occur when deploying multiple theories. 


Chapter 11 examines stakeholders’ views of AAU in the complex process of improving undergrad- 
uate STEM education. Chapter 12 begins to compare findings in earlier chapters to theories of 
scale, demonstrating how AAU’s work was similar to but also departed from the existing theories 
of scale. Chapter 13 reviews an area noted throughout our study as an important possible factor 
in scaling change — AAU changing its own culture to help foster changes in the campuses. 


Section Ill, the last section, offers conclusions and recommendations which distill lessons and 
ideas across the various chapters. 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 19 


Systems Approach to Change 


The Initiative worked at three major levels to encourage undergraduate STEM reform: the eight 
AAU STEM Project Sites, the AAU STEM Network, and national organizations in higher educa- 
tion. This three-level strategy demonstrates a systems approach in action. The Initiative lever- 
aged the interaction among the three levels to try to generate greater synergy among them, 
which is consistent with research about effective change strategies (Kezar, 2013). 


Individual AAU STEM Project Sites. The e/ght project sites were the major focus of the ini- 
tiative. As one respondent stated: “You need some action site where people can see the ideas 
are happening that you're trying to proliferate. So, these sites can become what people are 
talking about and help them to see and understand the AAU Initiative.” Unlike many prior efforts 
in higher education, the AAU Initiative emphasized institutional-level reform from the begin- 
ning. The Initiative based its framework and implementation strategies on the premise that scal- 
ing and sustainability required institutional as well as departmental and individual buy-in from 
the beginning, an atypical perspective in this type of reform. Most of the individuals interviewed 
perceived the AAU focus on institutional reform as unique and compelling. 


AAU used a variety of approaches to shape the commitment and direction of the individual cam- 
puses. AAU distributed the funding it received from the Helmsley Charitable Trust using a 
request for proposal (RFP) that required campuses to match funds, demonstrate commitment 
from their senior leadership, commit to a joint faculty and administrative planning and imple- 
mentation team, and to designate work that spanned the three levels of the Framework (ped- 
agogy, scaffolding, and cultural change). This approach to funding allowed AAU to retain lever- 
age over the change process after the monies were distributed because the expectations 
delineated in the RFP created a common vision for action and reinforced an institutional approach 
to change. This approach also created the opportunity for shaping the broader AAU STEM 
Network. AAU institutions that submitted unsuccessful applications still were influenced by the 
expectations set out in the RFP. This funding approach created synergies between the project 
sites and the AAU STEM Network. 


To help project sites develop projects in line with eventual institutionalization, AAU created a 
Framework (AAU Framework for Systemic Change in Undergraduate STEM Teaching and 
Learning) that described the three levels of change that applicants for the eight project sites had 
to address in their proposals: 1. Pedagogy — including assisting faculty in creating learning goals, 
assessments, and other practices that support new evidence-based teaching practices; 2. 
Scaffolding — including infrastructure, reform across curricula, and linking academic departments 
with support services that support new evidence-based teaching practices; and 3. Cultural 
change — commitment among senior leaders to the use of evidence-based teaching practices 
including revisions to the promotion and tenure process as needed. The Framework became a 
tool that could work across various levels of the higher education system, one that any campus 
across the country (including other AAU universities) could use to guide their change process. 
AAU reinforced the importance of an institutional and systems-wide approach to reforming 
teaching by using the Framework as a guide for interviews at project sites and for meetings of 
both project sites and of the AAU Network (e.g., concurrent conference sessions were organized 
in terms of pedagogy, scaffolding and culture change). 


3 There are other initiatives that have taken an institutional approach, particularly Project Kaleidoscope (PKAL) which has been 
focusing on the institution as the unit of change for over two decades. 
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The AAU STEM Initiative project team (also referred to as project leaders and Pls) also created 
and collected survey and metric data to help both AAU and project sites monitor progress in 
implementing evidence-based teaching in introductory STEM courses. As part of its larger dis- 
semination plan, the AAU Initiative project team wrote and disseminated Essential Questions and 
Data Sources including a list of survey ques- 

tions and related data to help other institutions The metric data reinforced the 


assess their progress in implementing under- synergy between the institutional, 
graduate education reforms. For more infor- 


mation see Essential Questions. The metric Network, and national actors in 
data reinforced the synergy between the insti- the change process. The metric 


tutional, network, and national actors in the . 
change process. The metric data provides an data provides an example of how 


example of how a tool developed within one a tool developed within one part 

part of a system — the institution in thiscase- of gq system—the institution in 

can have implications for the larger system. . é . 5 
this case—can have implications 


Project sites submitted annual reports to assist for the larger system. 

both institutional Pls and the AAU project team 

in monitoring progress. The AAU project team reviewed the reports and provided feedback to 
each of the eight project sites. The AAU project team led webinars for all project sites (and later 
for the AAU Network) to address key problems and strategies identified in the annual reports. 


AAU leaders created a variety of opportunities for the project sites to share information and 
learn from each other — annual meetings, webinars, grants for travel to other campuses, social 
networking sites to support communication and sharing of best practices (e.g. Hubzero/Trellis), 
a website with curated resources, listservs for the project teams, and workshops targeted for 
particular groups such as department chairs. Many of these strategies focused on the project 
sites but a few of them addressed the larger AAU Network. For example, the AAU website col- 
lected examples of successful innovations in teaching from all AAU Network members, not just 
from project sites. AAU also developed a communications plan that sent regular messages to 
AAU campuses about STEM education including grant opportunities, information resources, 
workshops and professional development, and state and federal policy updates. 


Site visits were conducted both at the beginning and end of the project. These site visits were 
meant to encourage and document implementation and scaling of educational reforms, not to 
act as an external evaluation of project work. AAU project leaders visited each project site to 
discuss its progress and meet with various stakeholders to re-energize the reform effort on that 
campus. AAU staff also offered to link campus Pls with individuals at other institutions doing 
similar reform work and to make presentations to campus constituents if requested by the teams 
at the project sites. 


AAU STEM Network. The AAU STEM Network focused on encouraging all AAU institutions 
to be “as excellent in teaching as research.” The RFP process showed a significant demand for 
a network: half of the 62 AAU members submitted proposals to become a project site. AAU 
worked with member campuses to appoint a point of contact at each campus. AAU sent infor- 
mation about the Initiative to each point of contact and invited them to attend annual network 
meetings where the eight project sites could interact with other AAU campuses. Information 
flowed both ways: many points of contact were successful reformers in STEM education and 
could help project sites refine their reform strategies. 
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Another mechanism to support the broader AAU Network was communication with key constit- 
uent groups in AAU, including presidents, provosts, and deans. The AAU project team, for exam- 
ple, periodically updated chief academic officers about the progress of the Initiative at their annual 
or semiannual meetings. The chief academic officers also were interviewed as part of each site 
visit by the AAU project team. 


AAU STEM Network sites also submitted examples of work on undergraduate STEM reform for 
AAU's sourcebook on the website, which helped further communication between network mem- 
bers. AAU staff encouraged collaboration between network members by mapping the activities 
happening on each campus and sharing the information across institutions. In addition, leaders 
in undergraduate STEM reform at non-project AAU campuses often served as expert speakers 
in AAU-sponsored meetings to inform the broader network. 


AAU staff forwarded information about potential grant funding for STEM undergraduate reform 
to all member institutions. AAU staff also helped non-project member institutions submit their 
proposals to other funding agencies. 


AAU also worked with the public affairs officers at each AAU STEM Network campus to discuss 
ways to publicize the efforts to improve undergraduate STEM education at each campus. For 
example, the New York Times published an article on the implementation of evidence-based 
teaching practices at the University of 


California, Davis. The University of North AAU used media attention to 


Carolina and Michigan State University were . oe 
showcased in an /nside Higher Education arti- help foster friendly competition, 


cle featuring the AAU STEM Initiative. Non- noting “Some campuses are 


project sites also garnered media attention. getting really good press for their 
For example, the University of Michigan's 


Center for Research on Learning and Teaching work to improve teaching, do 
and California Institute of Technology's Center you want to be left behind?” 

for Teaching, Learning & Outreach were pro- 

filed in a New York Times article on STEM reform efforts. AAU used media attention to help 
foster friendly competition, noting “some campuses are getting really good press for their work 
to improve teaching, do you want to be left behind?” 


Broader, National STEM Reform. At the national level AAU worked to shape the dialogue 
and activities about undergraduate STEM education. The Framework document and the Essential 
Questions and Data Source document were important means to spread the word about STEM 
educational reform on AAU campuses. Interviewees were familiar with these activities and gen- 
erally supportive of them. 


AAU also helped to support a coalition of national organizations aimed at improving undergrad- 
uate STEM education, named Coalition for Reform of Undergraduate STEM Education (Coalition 
or CRUSE). Coalition members include the American Association for the Advancement of Science, 
the Association of American Colleges and Universities, the Association of Public and Land-Grant 
Universities, Association of American Community Colleges, and the National Research Council. 
Linda Slakey, senior advisor to AAU, was the creator and now convener of the Coalition. 


AAU worked with the Coalition to map national-level STEM reform activities in STEM under- 
graduate education across associations and disciplinary societies. The goals of mapping were to 
coordinate work across national organizations and to institutionalize existing reforms. In the first 
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year of the Initiative AAU hosted a meeting of funding agencies and foundations to increase 
visibility of reforms and potentially lead to future funding of STEM undergraduate reforms. The 
Coalition convened the second meeting of 

funders and foundations, which led toa AAU mapped national-level STEM 


report called Achieving Systemic Change - A reform activities to coordinate work 
source book for advancing and funding 


undergraduate STEM education. Members of | @Cross national organizations and 


the Coalition were involved with AAU STEM to institutionalize existing reforms. 
Initiative meetings and webinars and worked 


with project sites as well as the AAU STEM Network. The Coalition also participates in the 
National Undergraduate STEM partnership established by the President's Council of Advisors on 
Science and Technology (PCAST). The Coalition represents a successful leveraging of AAU’s 
social and political capital with outside groups. 


Beyond the Washington D.C. area, AAU reached out to teaching and learning entities such as 
the Bayview Alliance (BVA); Cottrell Scholars; Howard Hughes Medical Institute (HHMI) 
Professors; Center for Integration of Research, Teaching and Learning (CIRTL), and Professional 
Organizational Development (POD). AAU invited these partners to attend meetings of the proj- 
ect teams and AAU STEM Network, which resulted in substantial information sharing between 
these organizations and project sites. AAU regularly communicated with the National Academy 
of Sciences (NAS) as well as STEM disciplinary societies, industry associations, and universities 
involved in STEM undergraduate education. AAU staff also took on leadership roles, sat on 
boards, and assumed positions on steering committees for the National Alliance for Broader 
Impacts, Business-Higher Education Forum, Accelerating Systemic Change Network (ASCN), as 
well as many related NSF project advisory boards. 


These partnerships successfully supported change at the national level. For example, following 
the AAU Initiative the Cottrell Scholars, a program devoted to improving undergraduate teach- 
ing in STEM, sponsored a collaborative research project to increase awareness among the STEM 
academic community about appropriate metrics to measure the effectiveness of teaching and 
to promote using these metrics in promotion and tenure decisions. These efforts have the poten- 
tial to help reshape evaluation of college teaching nationally. 


The advisory committee for the AAU STEM Initiative, which brought together esteemed individ- 
uals in the sciences as well as past and present presidents of AAU institutions, also helped pro- 
mote a national dialogue about STEM undergraduate reforms. The advisory committee influ- 
enced the strategies used by project sites and the messages communicated to the AAU STEM 
Network, providing yet another example of synergistic strategy across the multiple levels of the 
project. 


AAU is helping develop a national award to recognize college and university departments that 
have demonstrably improved undergraduate STEM education. Leaders at AAU spent years devel- 
oping the concept, working with potential funders and managing organizations to propose a 
process for offering and securing the award. Although the award would not be managed by 
AAU (or awarded only to AAU campuses) it is cast as part of the national effort to improve 
undergraduate STEM reform. 


AAU has historically been involved with advocating at the federal level for AAU institutions. 
Since the AAU STEM Initiative began, this advocacy has included seeking money from Congress 
to improve STEM education. AAU leaders also attended White House summits and the 
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working groups of the Committee on STEM Education (CoSTEM) to help coordinate federal 
investment in STEM education. They also worked with major foundations to develop a work- 
shop focused on increasing the funding for undergraduate STEM reform efforts from founda- 
tions. In this manner AAU influenced the shape of national policy and funding priorities as well 
as indirectly affecting AAU campuses that typically pursue such funding opportunities. 


AAU also had a robust public relations and media plan. AAU project leaders created media atten- 
tion through news articles about STEM reform efforts, by placing articles in major journals such 
as Science and Nature, and by attending national STEM reform conferences to promote the AAU 
STEM Initiative. AAU regularly issued press releases about the progress of the Initiative. These 
media efforts brought attention to STEM reform work in general and especially to the eight proj- 
ect sites. This media attention created a buzz among the broader AAU STEM Network. As an 
example, several interviewees cited the August 2015 Inside Higher Education article highlighting 
the AAU STEM Initiative’s efforts as well as snapshots of two project sites, the University of North 
Carolina, Chapel Hill and Michigan State University. Finally, AAU project staff presented at 
research conferences sponsored by the Association for the Study of Higher Education, and AAAS, 
and at the Transforming Institutions conference sponsored by Purdue University. 


In sum, AAU has had a substantial effect on the national policy dialogue about undergraduate STEM 
education. Many of AAU’s actions at the national level filtered down to its member institutions. Even 
so, interviewees from individual campuses were much less familiar with AAU’s role at the national 
level than they were with AAU’s work with individual campuses and with the AAU STEM Network. 
Perhaps more communication about AAU’s national work might create more synergy between its 
work at the national level with its influence on project sites and with the AAU STEM Network. 


Synergizing the Levels of the System. The 


Initiative leveraged the interactions among the Working across multiple system 
three levels to try to create and strengthen syn- 


ergy. The evidence suggests that AAU achieved levels helped AAU leverage Its 
such synergy in part by engaging organizations impact on STEM reforms. 

at the national level in a manner that assisted 

and/or influenced work within the STEM network and at the project sites. See Chapter 8 for more 
detail about this process based on networking. In this chapter we presented many examples of such 
synergy resulting from the use of the Framework document, metrics, RFP funding model, points of 
contact, networks, partnerships, and the measures of teaching excellence. 


The AAU STEM Initiative worked across and among the three levels — institution, AAU STEM 
Network, and national higher education environment. As predicted by institutional and systems 
theories, working across the various systems/levels helped AAU leverage its impact on STEM 
reforms. It was clear AAU took into account the differences among these three levels/systems 
and did achieve synergies across them. Yet AAU did not intentionally plan for synergies across 
levels/systems which potentially reduced the amount of synergy achieved in promoting reforms 
in STEM undergraduate education. We conclude that national organizations trying to maximize 
their effects on STEM undergraduate reforms should map and evaluate the different ways that 
the organizations are working across and among different systems. We highlight below other 
ways AAU could foster synergies among multiple organizational/systems levels: 

1. AAU might have communicated its work at the national level, particularly its important part- 
nerships, more effectively with individual campuses and the AAU STEM Network. Intentionally 
linking its national work (e.g., media attention, national partners like the Cottrell Scholars) 
with member institutions could provide synergistic support for on-campus work. 
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2. AAU might have used the members of the Advisory Committee to the AAU Initiative more 
directly in project work. Although the Advisory Committee reviewed project work and made 
general recommendations, it might also have played a more active role in advocating for 
change nationally. 

3. Although AAU offered several ways for project sites to share information, not as much 
effort went into sharing information between project and non-project sites. Providing 
mini grants for project sites to visit non-project sites is another way to leverage knowl- 
edge and communicate innovative ideas between organizations.* 

4. Interviewees often stated that they were unaware of the work occurring at these three 
levels, especially work in the broader national environment. Greater communication and 
articulation of AAU’s full strategy might have created more opportunities for synergy 
across AAU, its project sites, and the AAU STEM Network. 


Conclusion. This chapter examined how the AAU worked at three major levels — eight individ- 
ual project sites; the AAU STEM Network that involved all 62 AAU campuses; and the national 
level among higher education associations, national agencies focused on STEM education, busi- 
ness and industry partners, and philanthropic groups. 


4 AAU's current funding from Northrop Grumman includes funds for mini-grants for more campuses to take advantage of learning 
from each other. 
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Working the System 


This chapter explores how AAU can best position itself to influence its member institutions and 
the AAU Network in reforming undergraduate STEM education. Interviewees perceived that 
AAU was most effective in promoting STEM undergraduate reforms at project sites and across 
the AAU Network when it used three distinct strategies: 

= Strategies aimed at utilizing the prestige and influence of AAU 

= Strategies that built on the historic strengths of AAU 

= Strategies that emphasized AAU as a unique niche within U.S. higher education. 


According to interviewees, AAU was least effective when missing opportunities to leverage its 
identity with member institutions. Throughout this document, we will return to the importance 
of AAU’s unique position in the U.S. higher education milieu as a primary factor in its ability to 
influence member institutions and higher education writ large in the effort to improve under- 
graduate STEM education. In this context, networking is an especially powerful mechanism for 
AAU to take advantage of its position to influence institutions of higher education. 


We start with the national level of the higher education enterprise. Highlighted in this chapter 
are examples of where AAU has been successful in influencing the national conversation about 
STEM undergraduate education, as well as opportunities for increasing AAU’s effectiveness. 
Before proceeding, we note that the distinctions between the levels of the “system” are some- 
what artificial and not fully bounded. In describing its work, AAU staff did not separately define 
or discuss their activities in reforming STEM undergraduate education by level (institution, STEM 
Network, national organizations). For the most part, AAU staff viewed their work in the Initiative 
quite fluidly without distinctions by level or system. For analytical purposes, however, levels are 
important to help us understand the impact of AAU on local and Network reforms as an orga- 
nization that typically works primarily at the national level. 


Institutional theory suggests that innovations in organizations are strongly influenced by larger 
forces — beyond individual institutions — within an ecosystem. Several interviewees acknowledge 
that AAU’s multilevel or systems approach embedded in the Initiative helped foster change. As 
a director of a center for teaching and learning stated: 


“| think they [AAU] definitely play a prestigious role within the ecosystem that I’m 
embedded in. When they announced this initiative’s program, what | thought was 
interesting is how it motivated and enticed many of my colleagues at other institutions. 
Everybody was having this conversation. Hey, is your school putting one in? That 
wouldn't have happened if just any other organization said, ‘Hey, we're interested in 
doing reform.’ It was because it was the AAU. There's a certain level of ‘this will be 
something of importance to our university and to the landscape of the nation’s STEM 
teaching.’ There were messages coming over the POD network about this. So only a 
group like AAU can make that kind of national conversation happen.” 


This sentiment was common among interviewees: AAU was uniquely capable of shaping the sys- 
tem at these multiple levels. We explore the perceptions of respondents in greater detail below. 


Broader National Level. In this section, we describe respondent beliefs about where AAU 
could have the most impact at the national level. A few interviewees recommended additional 
activities in which AAU might participate. 
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National documents, resources and publications: AAU created several documents with 

a national audience in mind, including the Framework and the Essential! Questions and Data 
Sources documents. Collaborators in particular described the importance of these documents: 
“The framework will generate both conversation and provide directions for campuses interested 
in improving teaching in STEM courses, and this presents a very different perspective on how to 
go about this work®. This might be challenging to some but will definitely generate conversation 
because it comes from the AAU.” This comment (and many similar ones) reinforces the view of 
AAU as an information source with a built-in audience of interested local and national consum- 
ers of such information. One interviewee summarized this important role: “Any time something 
like this [Achieving Systemic Change document] comes out that has a lot of power. It can inspire 
people. They can give people ideas about how to do things they may not have thought about 
doing before. | think that’s a really good way to reach a broader audience and they could do that 
even more.” As an example of the type of influential literature envisioned in this quotation, in 
2017 (after this research study was conducted) AAU published /mproving Undergraduate STEM 
Education at Research Universities: A Collection of Case Studies. In general, respondents believed 
that resources and publications from AAU would be noticed by campus leaders across the coun- 
try. AAU can reach audiences of institutional decision makers such as presidents and provosts 
that STEM reformers typically cannot reach. In sum, by publishing and distributing national doc 
uments AAU took advantage of its unique national niche and organizational identity. 


Partnerships, networks, and convenings: AAU is uniquely placed to develop partnerships 
with a variety of organizations including the more elite, such as the National Academies of 
Science and National Institute of Health. AAU can also convene groups, such as funding agen- 
cies and foundations, that might not otherwise participate if invited by a less prestigious orga- 
nization. Interviewees appreciated the partnership and connections formed by AAU in support 
for reforming undergraduate STEM education: “I think there’s been some really important work 
[by AAU] with other organizations interested in advancing STEM, and it’s hard to work across 
organizations, | know that, but | think even more can and should be done to shape the national 
efforts and bring them together.” Several respondents asked even more of this activity: “I can 
imagine bringing together a group to talk about rewards systems at AAU institutions, or per- 
haps the value of professional development, or developing leaders within their STEM depart- 
ments or colleges, ways to evaluate teaching, or showcase major redesigned curriculum, or to 
discuss assessment of teaching excellence in STEM. Maybe convening some of the disciplinary 
societies, accreditors and industry to discuss curriculum, | can imagine this being a larger role 
in [AAU‘s] work.” 


Interviewees noted the importance of AAU’s role as a convener and suggested that it could 
expand that role. While AAU had done important work to convene the project sites and the 
AAU STEM Network, many interviewees suggested future convenings be focused on specific 
problems. Such topics might include changing tenure and promotion criteria or measuring 
teaching excellence. More focused meetings would allow for deeper deliberation about 
improving the undergraduate learning environment. 


Media and public relations: Many interviewees appreciated the high visibility of the Initiative 
in the media. As one administrator noted: “Media attention really helps shift priorities on our 
campus. AAU sending a press release or noting how a piece was picked up by a media outlet 


uw 


While it is too early to know the impact of the Framework and Essential Questions and Data Sources on the broader higher 
education landscape, it is likely that these documents will be utilized by a variety of campuses to help shape their STEM 
reform efforts. For other work, such as the national award, which has yet to be put in place, or work with the National 
Science Foundation, it may be too early to see its impact across the system. 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 27 


— that supported our efforts to innovate.” Many interviewees noted that AAU participation 
resulted in more media coverage for their campus (or other organization) than would other- 
wise have occurred. A collaborator commented: “Well, they are AAU — they can just draw more 
attention than others, they can open more doors at the media.” Respondents noted that their 
public affairs and relations staff cooperated with AAU’s public relations operation to increase 
media coverage of local efforts. 


National award or recognition for excellence in teaching: Early on, AAU recognized 
that a national award to recognize teaching excellence in STEM, particularly at the depart- 
mental level, might attract academic leaders’ attention and influence campus priorities. 
Respondents from various constituencies agreed with this assessment. As one example: 
“There just isn’t the same national awards or prestige for teaching. There are some awards, 
but one associated with the AAU will garner more attention.” Respondents believed that 
AAU was one of very few organizations whose sponsorship of a national teaching excellence 
award could achieve the visibility required to motivate large-scale change in STEM 
instruction. 


Advocacy: The AAU STEM Initiative is meant in part to help shape policy to improve under- 
graduate teaching and learning by interacting with federal entities such as Congress, the White 
House Office of Science Technology and Policy, and the National Science Foundation. 
Collaborators in particular felt that AAU’s track record with advocacy uniquely positioned AAU 


to leverage reform in STEM undergraduate ” 
education: “AAU needs to be at the table “AAU needs to be at the table for 


for the national conversations about STEM, the national conversations about 


about teaching, to make this a priority ; 
because them being there makes it a STEM, about teaching, to make 


priority.” this a priority because them being 

7 i i 33 
Teaching standards and measures of there makes it a priory: 
good teaching: Although respondents believed that AAU was not well positioned to alter 
reward systems on individual campuses, most believed that AAU could play an important role 
in defining standards or measures of teaching excellence at AAU institutions. One interviewee 
commented on the importance of AAU’s work with the Cottrell Scholars and related efforts 
in shaping the discussion about teaching excellence: “Given AAU is the standard bearer, they 
literally develop the metrics for defining excellence at AAU institutions. Creating those types 
of metrics around teaching would seem natural and legitimate and could really drive and scale 
change.” Although AAU has little history in defining metrics for effective teaching, for many 
years it has developed and reported metrics of research and scholarly productivity about its 
member institutions. 


Creating national level leadership with Advisory Committee and Coalition of 
Presidents: Many interviewees spoke about the importance of AAU’s national leadership 
in influencing local efforts to improve STEM undergraduate education. The AAU Initiative 
Advisory Committee and AAU member presidents are two groups capable of filling this 
national-level role. As noted by one respondent: “You look who's on their advisory commit- 
tee and those are people that can get the attention of national organizations, government 
agencies, disciplinary societies, business and industry — all sorts of key players. They could be 
leveraged more to get more media attention, to push various strategic initiatives.” Few other 
national organizations have the same influence in this regard as AAU. 
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AAU STEM Network. Interviewees also perceived AAU to have an important influence with 
various networks at the national level. They mentioned several ways to expand AAU’s influence 
in this regard: networks, influence, resource development, media, and convenings. 


Create and sustain networks aimed at STEM reform: Interviewees felt that AAU’s work 
with various networks and network development was among its most valuable contributions 
to reform: “AAU is well positioned to connect, convene, and has the cache to get people to 
show up for events.” We discuss these networking effects in more detail in Chapter 8. 


Work with Presidents and Provosts: Many respondents felt that AAU might involve its 
member presidents more directly in reforming undergraduate STEM education. As one exam- 
ple: “I’m not sure why there isn’t, or we aren't developing some kind of coalition of the AAU 
presidents who'd really get out there and are supporting excellence in teaching. We need a 
group of them to be champions and that would really help the efforts to scale change.” Said 
another respondent: “| believe that those parties [presidents and provosts] should have been 
engaged with the AAU Initiative from the beginning and much more directly. In other words, 
| mean, this initiative is on the one side and discussions among presidents and provosts tak- 
ing place in the other. We are told they hear about the Initiative, but | think there needs to 
be more of a continuous relationship taking place and feedback between the project sites 
and AAU president and provosts. And we need to tap the whole set of AAU institutions which 
can happen through those presidents and provost meetings.” AAU already regularly works 
with its member presidents and provosts so additional engagement in the STEM Initiative can 
be explored. We note, however, that AAU’s primary role is to represent its membership; it 
does not have the authority (nor does it wish to) instruct its member presidents and provosts 
how to participate in the Initiative. 


Points of contact: Respondents largely viewed the campus points of contact as underuti- 
lized in local STEM educational reforms. One faculty member comments: “! am not sure exactly 
our role —| think that has been evolving, but we could definitely be better involved, engaged. 
Many campuses that are doing some of the most forward-thinking work were not funded as 
project sites. And even since the funding occurred | have seen some other non-AAU campuses 
doing interesting [work] but it is not, there is not a good way to connect on that.” © 


Hold convenings: Interviewees consistently found the AAU national convenings useful to cam- 
pus reform efforts. They encouraged AAU to continue bringing together AAU campuses to dis- 
cuss ways to improve teaching practices. Some recommended convenings focused on issues 
related to the improvement of undergraduate teaching: faculty professional development, 
rewards structures, teaching and learning infrastructure, and finding ways to make teaching 
excellence a campus priority. As one interviewee stated: “AAU can bring together all the vari- 
ous constituents that are needed to create and scale change. Unless provosts and deans across 
the AAU are working together to address rewards it is not likely to happen one campus at a 
time. So, some things need to be tackled together through convenience. And scale can just be 
achieved better if professional development is discussed by staff working on that across AAU 


6 In fact, some of the AAU network sites that were not funded ended up making changes that were as large-scale and ambitious 
as the demonstration sites. These campuses that made changes were often striving AAU campuses that were interested in 
improving their reputation within the overall AAU and garnering more visibility. So rather than being motivated by the strategies 
of AAU, they were influenced by their own campus leader’s prestige-seeking within the network. The operative motivations 
that appear to be at play at the project sites and among the network focus more on prestige and leveraging prestige than 
other mechanisms that AAU leaders hoped would be in place such as learning from one another and planning processes 
to institutionalize change. In Chapter 6 on influence, we describe in more detail why AAU’s role may be more successful 
in scaling change when it leverages its influence and prestige rather than working closely at the project sites. 
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institutions, or deans having a group discussion about improving classrooms, the physical 
spaces, or curriculum models.” In sum, the ability to harness the power of networks to foster 
improvement in STEM undergraduate education was seen as a major strength of AAU. 


Media and competition among AAU campuses: |n addition to spreading the word about 
campus reforms, media coverage also fostered “friendly competition” to reform STEM under- 
graduate education. According to respondents, this type of peer pressure is a significant lever 
for reform. As one example: “When | saw activity happening on their campus it was a result 
of leaders having heard about the work on another campus and feeling they needed to cre- 
ate some more activity on their own.” Faculty and administrators both agreed that media 
attention and competition were strong sources of motivation for change on their campus. 


Resources: Interviewees viewed AAU as an important source of funding for project sites 
and for the AAU STEM Network, although the primary sources of funding for the Initiative 
were external organizations. AAU was also helpful in pointing member institutions to poten- 
tial funding sources for STEM Initiative-related work. In addition, AAU provided relevant 
resources such as metrics for teaching excellence. Respondents felt that resources should first 
be targeted to AAU institutions, but also noted how resources created through the Initiative 
could benefit the larger national higher education enterprise. 


Award or recognition among the AAU institutions: Although most respondents were not 
aware of AAU’s work to establish a departmental award for excellence in teaching, several 
respondents who were aware of the award enthusiastically supported the concept. For exam- 
ple: “there might be some kind of AAU award for innovation that they could provide, and it 
comes with a really powerful letter to the university’s administration when it’s given. You can 
imagine that [at] each of the reform locations they could identify an individual who did, X, Y Z 
and made significant contributions to A, B, C. That’s the thing that | think needs to happen.” 


Individual Campus Level. Evidence suggests that AAU was more effective in promoting 
reform in undergraduate STEM education by working at the national and network levels to influ- 
ence local reforms than by intervening at the campus level, at least in part because AAU did not 


have the resources to carry out extensive : = : 
on-site work. Working at the micro level to The primary difficulties for AAU 


interact with individual campuses is not the jn working with educational reforms 


most productive use of AAU’'s time and tth a | S ecataeaai’ 
resources. Since its beginning, AAU’s primary a ] Campus lever lS Mirnina 


relationship has been with university presi- Ongoing contact with individual 


dents and chancellors. It also has long-estab- faculty, instructional staff, and 
lished working relationships with vice presi- y : 


dents for research, graduate deans, and department chairs, as well as the 
university public/government affairs officers. local idiosyncrasies of how each 


AAU's relationships with individuals working campus makes decisions and 
on STEM education reform at each campus is 


more recent. AAU typically has not had the promotes (or does not promote) 

same type of working relationships with the use of evidence-based teaching. 
department chairs or faculty and staff directly 

involved with reforming STEM teaching on their campuses. In this context, AAU is limited in its 

direct influence on achieving institutional change. To increase its direct influence on the faculty 

and staff and department chairs implementing STEM reforms AAU would require additional 
resources and tools for working with local actors. 
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The evolution of the STEM Initiative demonstrates the complexities of working directly with indi- 
vidual campuses on STEM education reform. At the beginning of the STEM Initiative, AAU proj- 
ect leaders found local personnel confused about the role of AAU in the initial site visits. 
Personnel at project sites often assumed the visit was meant to evaluate project progress or rate 
each project. Instead, the purpose was to assist projects in their implementation. This lack of 
understanding was in part a function of the newness of this local role for the AAU. It took AAU 
project leaders much longer than they anticipated to develop rapport with members of the cam- 
pus teams. For example, the campuses were uncomfortable at first with collecting baseline data 
about teaching, including examining aggregate data across the eight project site universities. 
Developing trust between the AAU and the eight project sites required considerable work, 
although relationships between personnel at individual campuses and AAU project staff 
improved considerably over time. 


Interviewees’ comments acknowledged the limitations of AAU’s direct role in local reforms. As 
one example: “I’m not sure what they can actually do — for campuses. They are not set up that 
way or have those connections.” As another example: “I think it is mostly just who they are - 
who they can influence, not anything they can do working with campuses.” 


Our data show that AAU, as is true of many national organizations for higher education, is not 
set up to work on an ongoing basis directly with individual campuses as they attempt to reform 
undergraduate STEM education. AAU has much closer ties with senior administrators on tis 
member campuses than it does with individual faculty and staff working in STEM education. As 
we discuss throughout the remainder of this report, the primary difficulties for AAU in working 
with educational reforms at the campus level is minimal ongoing contact with individual faculty, 
instructional staff, and department chairs, as well as the local idiosyncrasies of how each cam- 
pus makes decisions and promotes (or does not promote) the use of evidence-based teaching. 


These findings demonstrate that AAU appears better-positioned to impact change at the net- 
work and national levels than at the individual campus level. Placing even more emphasis on the 
network and national levels may advance its efforts going forward. AAU appears most effec 
tive when focusing on coordinating with national groups, helping shape national priorities and 
funding, garnering media attention and influencing campus administrators’ perspectives and 
priorities. Chapter 6 elaborates on the nature of AAU as a prestige organization and the conse- 
quences of its actions for scaling change in STEM undergraduate education. 


Leadership for Scaled Systems Change. Interviewees suggested that the Initiative so far 
had not spent many resources on leadership development, which they saw as a missed oppor- 
tunity. Below we examine interviewees comments about leadership. 


Individual project sites: Almost everybody interviewed described the importance of strong 
leadership to institutionalization. Faculty leadership was necessary to champion new pedagog- 
ical approaches in departments and academic leaders’ (chairs, deans, provosts) support was 
needed to develop policies and structures to support teaching quality. According to respon- 
dents, most project sites lacked meaningful leadership at one or both levels for some time. 
Administrative turnover adversely affected reforms at some campuses. At some campuses no 
mechanism existed to continue institutional resources for STEM reforms after the project fund- 
ing ended. Some respondents sought to identify roles for leadership at different levels to sup- 
port STEM teaching reforms: “| don’t think we really know what senior administrative support 
and commitment really looks like. We talk about it but, | saw this in this effort, and | have seen 
it in others. We just do not hold administrators to a particular set of activities or tasks, and | 
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saw this as a gap in this initiative as well.” At the two campuses where the Initiative was insti- 
tutionalized, senior administrator commitment and support was present. For example, at one 
campus, interviewees noted how the chancellor and provost were extremely knowledgeable 
and committed to the Initiative; they have created a new role of vice president of instruction, 
built new active-learning classrooms, identified funding to support the reforms in the future, 
and connected the Initiative to their broader institutional processes such as accreditation and 
strategic planning. Both faculty and administrators on campus identified these as signs of insti- 
tutional commitment and senior administrative support. 


Ensuring Multi-level Leadership: The development and connection between leadership 
at multiple levels (e.g., bottom up from faculty, in the middle with deans, and fro the top with 
the president and provost) is important to scaling STEM undergraduate reforms. On some cam- 
puses even when one or more leadership levels supported the reforms the linkage between 
them was not evident. On one campus, the Initiative was too top-down; the provost submit- 
ted the proposal with limited faculty and departmental involvement. At another, key faculty 
members and top administrators were supportive but the buy-in from some department chairs 
and other faculty members was less. AAU emphasized the importance of leadership at multi- 
ple levels in its RFP but probably needed to help project sites develop and monitor the roles of 
leadership in the reform process. Perhaps developing a “check box” matrix with involvement 
by different forms and levels of leadership would help local campuses monitor involvement by 
leaders in STEM undergraduate reform. 


Articulating Senior Level Leadership: AAU potentially has more direct influence on its 
primary constituencies, presidents and provosts. AAU might explore options for encouraging 
participation by these leaders in STEM undergraduate reform. One campus provided an exam- 
ple of what this type of senior leadership might look like: 


“To give an example of what | think AAU could provide, at a recent meeting of the 
faculty, the dean of the College of natural science came to talk to the faculty about 
Gateway course reform. In fact, one of his associate deans gave a presentation, 
and the associate dean mentioned up front and center — and he put a picture of 
the university president up on the screen — and says this is a priority for the entire 
university. There’s a list of the key projects the university is committed to with the 
AAU listed front and center. Part of the whole point of the presentation was to 
say to the faculty, the president and this national organization that the university is 
involved with think this is really an important topic. The dean thinks it’s so important 
that he’s right here in the room with you wanting to talk to you about this. What are 
you as a department going to be doing to further Gateway reform? There was some 
discussion of particular modifications of courses and so on that proceeded. And the 
dean, all the way down to the faculty and the room was really engaged. There was 
mention of outside resources from these grants and then there’s money that the 
provost and the dean who provided, just showing that this is all flowing in the 
same direction. You should get with this direction because it’s really important 
to all these individuals and groups.” 


This interviewee felt that AAU Initiative project staff could offer more tangible examples to 
guide actions by senior administrators in reforming undergraduate STEM teaching and learn- 
ing. AAU might provide information to help local project leaders—the “champions” on each 
campus—to find ways to build both top-down and bottom-up support for reforms, including 
identifying appropriate roles of different types of leaders in those processes. 
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Build leadership skills of AAU STEM Network champions: Based on the success of 
early convenings of individual campus project staff by AAU, many respondents recommended 
that AAU expand its role to offer a Leadership Institute for champions of STEM education 
reform. The Leadership Institute could allow AAU campuses to engage deeply with the 
Framework, discuss leadership development on their campuses, and find ways to institution- 
alize successful reforms. 


National level leadership: Finding the right leadership niche within the Coalition for 
Reform of Undergraduate STEM Education. AAU can also play a larger role in creating 
collective impact in working with the Coalition for Reform of Undergraduate STEM Education. 
Even though AAU is relatively new to undergraduate teaching reform, unlike many other coa- 
lition members it can garner attention from such key organizations as the National Research 
Council, the National Science Foundation, the White House, Congress, and the National 
Academies of Science more effectively than most members of the coalition. In general, inter- 
viewees felt that AAU involvement at the national level was important to further the reform of 
undergraduate STEM education. 


Conclusion. This chapter highlights interviewee perceptions about the best roles for AAU in 
scaling changes to undergraduate STEM education. Especially important, according to inter- 
viewees, was AAU’s influence at the national level where its history and prestige were most 
effective in supporting reforms. 
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Leveraging Influence for Change 


Chapter 4 described how the AAU STEM Initiative has shaped systems change at various levels. 
Chapter 5 reviewed what participants said was AAU's strongest role in that system, its prestige 
and influence. In this chapter, we examine in-depth how AAU leveraged its prestige and influ- 
ence to encourage change. We also discuss missed opportunities that might inform AAU as it 
continues with its work on the AAU STEM Initiative. 


Interviewees said that AAU’s leveraging of its influence was the most important role it played in 
creating change. Elite campuses as a collective, i.e, AAU, and individually strongly institutions 
influence the broader system of higher education including its values and ways of doing work. 
For example: 


“When Harvard changed its medical education — then all the others followed.” 


“| think one thing that was striking was, on a positive side, just having the name of 
AAU associated with it, which is a very prestigious organization. Seeing a prestigious 
organization that had not previously been a giant player in this arena, shall we say, 
choose to step in, that was really important.” 


As noted by a plenary speaker at an AAU conference on cultural change: “culture eats strat- 
egy for lunch.” This statement is consistent with the findings of this study and others: when 
seeking to reform instructional practices the values and norms of universities are more influ- 
ential than change strategies. Focusing on 

shifting culture, especially in departments and Interviewees said that AAU’s 


academic programs, vis-a-vis reinforcing par- leveraging of its influence was 
ticular values and priorities is fundamental to 


implementing and especially institutionalizng | the most important role it played 
reforms in undergraduate STEM education. jin creating change. 

This chapter examines ways that the Initiative 

fostered attempts to influence values and priorities, alter norms, create institutional logics, and 
leverage networks. Influence, in particular, is an important contribution to the literature which 
typically has emphasized technocratic strategies such as planning, data, and communication. 


The AAU Initiative gave less overt or intentional attention to influence strategies, creation of 
new norms/values, and networking and more attention to strategies such as data collection, 
information sharing, or resource identification. AAU did consider how influence needed to be 
balanced with ownership of the Initiative, which is important to effective use of influence strat- 
egies. Even if somewhat unintended, AAU’s technocratic strategies such as resource identifica- 
tion were ultimately in service of influence, resetting norms, or improving networking. 


Utilizing Influence to Push for Change in Values and Priorities. By establishing the Initiative 
AAU took advantage of its most important role in reforming undergraduate STEM education, indi- 
cating to member institutions the importance that AAU placed on enhancing the value and prac 
tice of undergraduate teaching. AAU serves as the “gold standard and standard bearer for the 
entire system.” Almost all interviewees found the Initiative alone an important factor in change: 


“The simple existence of this initiative and the fact that AAU is really an organization of 
the top leadership, of the presidents and the provosts. They [other campuses] seem to 
be getting the message. As | talk with universities, they’re very much aware that this 
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initiative is going on. Even if I’m talking at a research university that’s not specifically an 
AAU university, AAU is — | mean that’s the gold standard in terms of a research 
university so that many of the other universities, especially the large publics that aspire 

to become AAU members, look at this effort and decide [that] trying to promote 
improved STEM education at the undergraduate level is something that’s worth pursuing.” 


Interviewees found the general message of the Initiative more important than specific strate- 
gies or change approaches: “It’s more of a general awareness than specific project activities 
that they're [other campuses] trying to emulate.” 


At nearly every meeting we attended, we heard similar sentiments: AAU was able to capture 
the attention of institutional leaders and was therefore capable of influencing campus behav- 
iors. At a department chair workshop that we attended, the first comment made by a faculty 
member to the AAU was: “You have my institution’s attention — they want to be an AAU insti- 
tution and care about that. So that is what can convince our campuses. You are influential.” 


In addition to leveraging its prestige, many respondents noted the power of AAU as a high- 
stakes organization. AAU can eject member-universities if they do not align with AAU values, 
which enhances AAU’s ability to influence its members. 


Influencing whom? Interviewees described a variety of groups and organizations with which 
AAU holds sway, and discussed the degree to which the Initiative was successful in influenc- 
ing those groups to date.’ 


Prestigious, national higher education players. First, interviewees noted AAU’s influence on the 
other prestige groups that shape higher education, including the National Academy of Sciences, 
National Science Foundation, the National Research Council, Howard Hughes Medical Institute, 
and Congress. These organizations shape the environment for STEM — its funding, policies and 
priorities. One interviewee describes this impact: “There are few organizations that are presti- 
gious enough that they can make Congress listen. They can make the National Science 
Foundation listen. That means they [AAU] can make things happen.” AAU was described as 
one of the few groups that could get the attention of organizations that set national priorities, 
funding, and direction. One interviewee commented: “Imagine if the National Academies made 
teaching a priority, a real priority, that could change things. AAU can work with them [National 
Academies] and others — setting each in this direction to improving teaching.” 


AAU campus leaders. Many Interviewees also noted AAU’s influence on senior campus admin- 
istrators — presidents, chancellors, and provosts — at their institutions, particularly in setting pri- 
orities for the campus: “AAU has the presidents’ and provosts’ ears. Our initiative was men- 
tioned at a meeting of presidents and the next day | got a call asking about our initiative. And 
| know that is just one example, but | have other examples of when AAU has made an issue a 
priority and then it comes down the pike at our campus.” 


Leaders at other campuses. Interviewees found AAU’s influence on senior administrators to 
extend beyond AAU institutions. One administrator described the influence that AAU has on 
presidents and provosts across the country: “AAU is an organization that’s known by all 


NSN 


Some faculty admitted that they had never heard of the AAU before the Initiative, but have since learned that it is an 

important organization. They said, however, that their faculty colleagues are largely unaware of what the AAU is and its influ- 
ence. Thus, AAU did not have as much influence on some individual faculty members at AAU campuses. This issue is detailed in 
Chapter 11. 
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presidents and provosts across the country, so certainly it can impact presidents of AAU insti- 
tutions, but it’s also shaping the perspective of leadership on all sorts of campuses across the 
country.” AAU influenced the broader array of colleges and universities in two ways: (1) shap- 
ing institutions that want to become members of AAU and (2) peer pressure by emphasizing 
reforms in institutions that define excellence in U.S. higher education. As one administrator 
described: “There’re a lot of wannabes out there. And this sort of initiative not only impacts 
the AAU campuses but those that are striving to become an AAU campus. They're always look- 
ing for what's happening at AAU campuses. So, there’s the potential there to impact all of 
these campuses as well.” Others described peer pressure: “If AAU campuses start to teach 
STEM quite differently, this will create pressure for other institutions to teach in new ways — 
they will not want to be perceived as too far from the standard.” AAU can leverage its status 
as a Standard bearer to create pressure for change throughout the system — an important com- 
ponent of scaling change. 


STEM reform community. Several organizations and initiatives focus on STEM reform — 
Bayview Alliance, CIRTL, PKAL, disciplinary societies, grant funded projects from NSF, and the 
like. These groups help shape the dialogue about future directions for STEM. Interviewees 
described opportunities for AAU to influence and benefit from working relationships with 
these other organizations: “There are these other initiatives but many of them are not cen- 
tered in research universities and those that are could still use the support and influence of 
AAU. So, there can be a lot more done to leverage AAU’s influence to forward the efforts 
of CIRTL, for example.” In particular, AAU can push the boundaries of excellence in teaching 
and promote the work of these other organizations to advance STEM reform in general. 


More influence needed: Respondents felt that AAU should collaborate more systemati- 

cally with the groups that shape the national dialogue about reforming STEM undergradu- 

ate education: “| have to admit | don’t know the work they may be doing with these groups, 

but the fact that | don’t know a lot about activities they’re engaged in suggests they could 

be doing more or making the work better known.” Another expressed a similar sentiment: 

“| don’t get the sense they are doing 

enough to make campus leaders— whether “I don’t get the sense they are doing 


it be AAU presidents or leaders at other enough to make campus leaders 
campuses — to make this a priority. Leaders 


across the country should really be ques- .—-Whether it be AAU presidents or 

tioning how good we are at teachingwhen Jeaders at other campuses-to make 

there are these indictments coming out in . — 

national sludies and reports” this a priority. Leaders across the 
country should really be questioning 


Often, we found that respondents were unaware how good we are at teaching when 
of the existing collaborative work between 


AAU and other important organizations. We there are these indictments coming 


observed many of these collaborative efforts. Qut in national studies and reports.” 
So, part of the problem may be communication 

as much as action. Even so, respondents were quite unified in their belief that AAU’s ability to 
influence these important organizations was among the most efficacious strategies for creating 
change in STEM undergraduate education. We also encourage AAU to consider alternative ways 

to hone its influence on external groups. As one example, Hunter Rawlings, then president of 

AAU, helped spread the word about educational reforms when he asked campus presidents if 

they knew how their institution was responding to the AAU STEM Initiative. Turning unofficial 

events such as this one into more formal strategies might expand AAU’s impact on STEM reforms. 
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Consistent with institutional theory, the more AAU works with and influences a variety of groups 
to form a single dominant logic about STEM undergraduate reforms, the more likely change is 
to occur on college campuses. We explore the relationship between influence and institutional 
logics in more detail in the next chapter. 


Approaches for Influence and Leveraging Prestige. This section discusses the approaches 
for AAU to leverage influence most often mentioned by interviewees and observed by the 
research team. 


Name-dropping: Many interviewees (project sites and points of contact) talked about using 
the name of AAU or its member institutions to persuade individuals on their campuses to 
consider new instructional practices. Here is how one faculty member described this connec 
tion: “When you're making the case to faculty who are very focused on their own research 
agendas, the ability to name drop other institutions is very helpful. To be able to say, ‘| know 
from the AAU STEM Network that Duke or Brown is doing this.’ The ability to have a little 
bit ‘of our peers are doing this, we ought to be doing this’ — that has been enormously help- 
ful.” One point of contact described the way she used the AAU name with senior campus 
administrators “Having the Initiative just exist has helped me in recruiting more support for 
new classrooms and money for professional development. | mention the Initiative and what 
| learned at a recent meeting and resources flowed.” In sum, using the names of AAU and/ 
or its member institutions was a useful strategy for generating buy-in among faculty and 
administrators for undergraduate STEM reforms. 


Branding as an AAU Project Site: Project sites that actively leveraged the AAU name on 
campus seemed more likely to be the most successful projects. Some of the project teams 
called themselves an “AAU” project and used the AAU name on all of their communications. 
Interviewees found that this branding increased visibility on campus and generated greater 
administrative support. 


Competition: Competition among AAU institutions is a core part of being a member. 
Leveraging the competition assisted some project sites in their work. As one administrator 
noted: “There's just this built-in competition between AAU institutions. They'll want to be in 
the top tier and are competing constantly to improve. So that competition was built into this 
initiative - we competed to be project sites, as project sites we compete to contribute to the 
initiative, and we want to be perceived as leaders in this work.” During interviews, the points 
of contact described how they positioned themselves for future competition in the AAU STEM 
Initiative. As one faculty member noted: “I know our campus leaders are very interested in 
competing for funding if it were to become available from AAU. There’s also a perception 
they don’t want to be too far behind in terms of changes. So that has helped get support for 
the work we're doing on our campus.” Another quipped: “The proposals in the creation of 
those centers had impact because in a way, everyone wanted to be the one doing the best 
thing. | guess this friendly competition is the catalyst of any changes.” This competition 
between sites would likely not arise in a less influential organization. 


AAU peer group affect or comparison: Many interviewees noted that AAU also lever- 
aged change by emphasizing the similarities between member institutions. Having peer insti- 
tutions advocating and demonstrating instructional reforms is quite effective. As noted by 
one campus leader: “We really are driven by meaningful comparison groups — ones that are 
similar, that seem to have similar goals and environments ... so if your peers are doing some- 
thing and you're not, that brings some pressure to bear on you especially if it’s something 
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that you believe is important. And so, | think the involvement of the AAU provides motiva- 
tion to our universities and to all universities who want to be an AAU institution.” Whether 
described as a “comparison group” or “peer group,” interviewees found their faculty and 
administrators much more likely to engage in change processes when working with peers. 
Interviewees provided counter examples where working with other types of institutions was 
less effective: “Our campus has been involved with a lot of other STEM reform efforts but 
none of them have had the kind of traction this initiative has had. From everything | can 
observe, it’s clear that is because this involves institutions that are research-oriented, every- 
one is much more interested and there are more changes going on that I’ve seen as a result 
of other initiatives.” 


Interviewees found that the legitimacy of AAU makes its advocacy for the use of evi- 
dence-based teaching in STEM more effective in gathering support from faculty and admin- 
istrators than is research demonstrating the benefits of active teaching and learning. One 
administrator described this phenomenon: “On my campus, other AAU campuses are much 
more influential than anything else — studies or evidence from NRC [National Research 
Council], NAS, disciplinary groups or others that relate or reach out to non-AAU type insti- 
tutions. Unless the message is customized to our peer group, it isn’t received.” 


External recognition and reward: Several individuals favored AAU or an equally promi- 
nent organization supporting formal recognition to campuses (or departments) with demon- 
strated high quality undergraduate teaching. Departments or institutions vying to be recog- 
nized for their teaching is a useful form of competition in scaling STEM education reforms. 
One interviewee commented “In terms of influence, an award would also help. It builds on 
our inherent sense of competition.” As noted earlier, AAU has been working to develop a 
national award to be given to departments demonstrating excellence in teaching. 


Media and press: |nterviewees also mentioned AAU’s work with the media and press as 
influential on their campuses. One administrator stated: “My president said to his cabinet — 
‘| need to know more about this initiative on my campus, because there’s been a recent arti- 
cle about it in the press.’ Everyone then was contacting me and that got a lot of attention 
and support and furthered my work on campus.” Interviewees noted how media coverage 
is especially important in influencing presidents and provosts. 


Site visits: Site visits by AAU project staff helped campus project teams gain credibility on 
their campuses. During site visits the AAU staff met with deans, provosts and often presi- 
dents, which enhanced project visibility at senior leadership levels. A faculty member com- 
mented: “Those site visits. They really get the attention of our provost and deans. You can 
see changes taking off after the AAU team has come to campus.” Site visits also helped pro- 
mote communication of local project activities both across project participants — which 
helped within-project communication — and to faculty and administrators not currently 
involved with project activities. Interviewees found the site visits especially helpful in gain- 
ing support from the latter group which often were influenced by the presence of AAU. In 
sum, the AAU project team site visits supported reforms on campus both because they 
helped cement the support of senior leaders and because they helped extend the reach of 
local campus projects to individuals who otherwise might not have become engaged in or 
aware of the campus reform efforts. 
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Partnering with other influential organizations: |nterviewees noted that AAU was most 
effective when it partnered with other prominent national groups to promote effective under- 
graduate teaching in STEM. Various respondents indicated the importance of having these 
national groups — including funding agencies, National Research Council, NSF, among others 
— in attendance at AAU STEM Initiative meetings. One collaborator describes this impact: “1 
have talked a lot with leaders at the AAU campus sites and | can tell that having HHMI or 
Helmsley [Charitable Trust] here at the meetings, that makes people pay attention more and 


| think they have more confidence that in . : 
five years they're still going to be people “I have talked a lot with leaders at 


talking about improving teaching in STEM.” the AAU campus sites and | can tell 
Balancing influence and being a trusted that having HHMI or Helmsley here 


partner: The AAU project team recognized at the meetings, that makes people 
that playing up AAU's power with its mem- pay attention more...” 

ber institutions — evident during the initial 

contacts with member campuses — could adversely affect reform efforts. From the beginning, the 
project team believed that long-lasting reforms of STEM education were more likely when local 
faculty members and administrators “owned” the reforms. If project sites viewed the AAU 
Initiative as a top-down directive instead of as a partnership with AAU the likelihood of partici- 
pation, effectiveness, and scaling at project campuses would be reduced. As a consequence, the 
AAU project team sought to become trusted partners in the reform process rather than acting 
as an outside agency trying to impose policy change. The AAU project staff carefully monitored 
the campuses’ suggestions and reactions to their work. Project campuses, as one example, 
reviewed AAU survey documents and made regular suggestions about the Initiative as it unfolded. 
As another example, the AAU project team specified the types of individuals it wanted to inter- 
view during site visits but ultimately left the choice up to project sites. We will return to the con- 
cepts of balancing and negotiating different theories of action in Chapter 10. 


For the most part, interviewees uniformly found AAU to be a collaborative partner, one that did 
not overstep or abuse its power. Interviewees described AAU'’s influence in the Initiative in a 
positive way; even critics of the Initiative noted that AAU used its influence effectively without 
imposing, dictating, or forcing actions. Interestingly, most of the critics wanted AAU to be more 
aggressive in asserting its influence on individual campuses. 


Additional Areas for Influence. Although most interviewees felt that AAU’s most effective 
role in fostering STEM reforms was influence, they were often uninformed about the degree to 
which AAU was or was not active in seeking to influence STEM reforms. In some cases, inter- 
viewees recommended that AAU expand its influence in ways that we knew AAU had already 
adopted. It may be that AAU requires a bit less additional effort in influencing STEM reforms 
and more communication about what it already is doing. Below we describe recommendations 
by respondents and our assessment of them. 


Even more partnering work with influential organizations and groups: Although inter- 
viewees recognized AAU’s efforts to influence what are termed “field actors” in institutional 
theory — groups that can frame the landscape or system — many felt that AAU might give more 
attention to this task. Organizations mentioned by respondents as missed opportunities for part- 
nerships included accrediting agencies, textbook companies, leaders in the assessment move- 
ment, individuals involved with the MCAT revisions, and the National Science Foundation. One 
interviewee mentioned how the National Academy of Sciences has no award or recognition for 
teaching. They went on to indicate that if the NAS changed their criteria to include excellence 
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in teaching it would create an elite teaching society and create a sustained effort to take seri- 
ously teaching reforms. 


It is important to note that AAU already was working with the American Council on Education 
on federal education policy, as well as with the Association of Public and Land-grant Universities, 
regional and national institutional accrediting agencies, disciplinary societies, the National 
Academies of Sciences, various private foundations and funding organizations, the National 
Science Foundation and White House Office of Science and Technology Policy. Many of these 
working relationships were not communicated by AAU to its member institutions or at least to 
many of the faculty and administrators working in them. 


Leveraging their prestige and influence for change within the national or macro 
system: |nterviewees noted how change efforts seem focused primarily on the AAU proj- 
ect sites. The STEM Network was an effort by AAU to expand the Initiative to all member 
institutions and implicitly to leverage AAU’s prestige to effect the larger national higher edu- 
cation environment. Some respondents recommended making the national implications of 
the Initiative more visible: “I think it’s that idea, the very fact that AAU represents these 
highly select research institutions and we are interested in STEM education. It’s saying to 
not just our institution but to higher education as a whole that this is really important, and 
that we need to make changes. We, the AAU, believe this is important enough that they're 
going to spend their capital and their time and effort on thinking about STEM education 
and getting these research institutions to think about STEM education and make changes 
in STEM education.” 


Expanding membership criteria: Several respondents advocated for AAU including teach- 
ing excellence as a criterion for membership. As one example: “Campus leaders are really 
focused on the AAU criteria and those are all focused on research. | am just not sure how 
much progress will actually be made if the criteria are not changed and I'm very convinced 
that campus leaders will take note if teaching is a part of the AAU criteria.” As we discuss 
in more detail in Chapter 13, this sentiment expresses the importance of AAU’s endorse- 
ment of quality teaching. The sentiment, 
however, masks the ee of both the “Campus leaders are really focused 
way that AAU sets its criteria for member- On the AAU criteria and those are 


ship and the metrics available for measu- all focused on research. |am just 
ing teaching excellence. The criteria for 


AAU membership is not a decision made Not Sure how much progress will 
solely by AAU staff. It is an expression of actually be made if the criteria are 


the general membership as represented by F : 
the presidents of AAU institutions. Equally not changed and I'm very convinced 


complex and perhaps more problematic is that campus leaders will take note if 
the lack of available metrics to easily mea- teaching isa part of the AAU criteria.” 
sure teaching excellence especially for insti- 

tutions as a whole. Although expanding membership criteria to include teaching quality 

seems a Straightforward way to encourage the improvement of teaching at AAU institutions, 

enacting such a change to membership criteria is much more complicated than implied by 

some interviewees. It is an area worth further investigation though. 
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Setting standards around teaching and evaluation of teaching: Many interviewees 
recommended that AAU endorse a set of standards for teaching quality. Interviewees 
acknowledge that AAU could not act alone. AAU might partner with groups such as the 
Cottrell Scholars to develop some national benchmarks that might help shape institutional 
efforts in evaluating teaching. As one collaborator noted: “As a national organization you 
[AAU] can call attention to issues or problems like the PCAST report did, but you [AAU] can 
also outline new ways for people to think about work. | think AAU has a real opportunity to 
define for research universities what quality teaching is in that environment and | think they 
need to extend themselves even more in this area.” We note that some resources already are 
available. AAU’s Essential Questions and Data Sources is one of them. The recently published 
(2017) Indicators for Monitoring Undergraduate STEM Education by the National Academies 
of Sciences is another one. 


Challenges to an Influence Strategy. 


Maintaining priority from afar: Respondents raised questions about how to sustain AAU 
influence over time. Some interviewees emphasized the episodic nature of AAU influence. 
AAU's presence was elevated when it announced the Initiative, during the awarding of proj- 
ect sites, and (for grant recipients) during the site visits. In between these times the influ- 
ence tended to fade. Other interviewees suggested that episodic engagement by national 
organizations such as AAU is natural. AAU has many tasks to perform and limited resources 
to perform them. To these interviewees episodic engagement need not adversely affect the 
success of the Initiative. One interviewee described the issue in this fashion: “Il guess the 
AAU recommendations end up—and even if not directly influencing the process, | guess 
they lurk in the background.” She was noting that AAU is most influential not directly on 
an ongoing basis, but indirectly, as a value system lurking in the background. Feedback about 
the efficacy of episodic and distant influence was perceived differently by stakeholders. Some 
stakeholders felt a national organization could create mechanisms for change (e.g., adding 
teaching quality to AAU membership criteria) while others felt that AAU could not monitor 
reforms on an ongoing basis. For these respondents, AAU’s primary contribution was to 
leverage its influence to encourage member institutions to work toward improving under- 
graduate STEM education. 


Interest alignment: Institutional theory hypothesizes that change occurs when influential 
groups within the sector align their messages and policies. Such alignment is challenging in 
the AAU Initiative because so many initiatives with their own imperatives are underway at 
the same time. These include the Center for the Integration of Research, Teaching and 
Learning (CIRTL), Network of STEM Education Centers (NSEC), PULSE Vision & Change, and 
Bayview Alliance (BVA). Each initiative, including AAU’'s, relies on different leverage points 
and approaches to creating change. They even have different priorities in the change pro- 
cess. CIRTL, for example, emphasizes the training of prospective faculty members to use 
active learning strategies, whereas AAU is trying to influence current instructors and their 
academic departments. These initiatives also vary by level of focus with, for example, some 
centering their efforts on the department and others on institutional consortia. Interviewees 
noted the complexity of the collection of STEM reform initiatives and the relative lack of 
coordination between them. As one interviewee noted: “These various efforts — sometimes 
they just seem to have different methodologies. Their work is not at odds with each other, 
but rarely is work connected meaningfully.” 
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Conclusion. The consensus among interviewees is that AAU’s most effective approach to 
change is leveraging its own influence and prestige. Successful influence strategies included 
competition, setting up comparison and peer groups, public relations and press, and national 
awards and recognition. AAU project and network sites leveraged the prestige and status of 
AAU to foster reform on their campuses. Interviewees had some specific recommendations for 
improvement but for the most part, interviewees struggled to detail these alternatives. 


The research team makes the following recommendations: 1. Identify and map the groups/orga- 
nizations on which AAU is likely to have the most influence including but not limited to the list 
of groups discussed in this chapter; 2. Consider strategies for working with the groups that build 
off of AAU’s capacities, unique strengths, and assets; and, 3. Communicate its influence work 
with various groups when such communication would not jeopardize the very influence AAU is 
trying to exert. 


Finally, we note that in some situations communicating or making visible efforts to influence 
other groups and organizations would be counterproductive. It is quite possible that AAU could 
not always be forthright in describing influence efforts as the process of influencing could be 
compromised by such communication. Even though influence is a complex change strategy, 
designing an effective influence strategy is a key to AAU’s success in its Initiative to improve 
undergraduate STEM education. 
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Recreating Institutional Logics/Norms 
to Frame and Message the Change 


After influence, respondents mentioned AAU’s role in reshaping norms and values most often. 
According to institutional theory, norms and values can be altered by changing institutional log- 
ics (Scott, 2008). Institutional logics are the “socially constructed, historical patterns of material 
practices, assumptions, values, beliefs, and rules” through which people attribute “meaning to 
their social reality” (Thornton & Ocasio, 2008, p. 804). Logics are embedded in structures, prac 
tices, and the cultures of institutions. Logics indirectly govern the behavior of actors and groups 
of actors within organizations by dictating the “sources of legitimacy” of particular ideas 
(Thornton, Ocasio, & Lounsbury, 2012, p. 56). Behaviors among individuals in organizations are 
thought to be “a consequence of taken for granted beliefs, schemas, and ideas that originate 
in larger institutional contexts” (Leicht & Fennell, 2008, p. 2). 


Institutional logics are identified through values and rationales described in the language and 
discourse of important entities (e.g., associations, accreditors, legislative bodies, National 
Research Council) within the organizational field (Thornton & Ocasio, 2008). A field can be 
guided by a single logic, or multiple and/or contradictory logics. For the last hundred years, the 
predominant institutional logics driving research universities, particularly elite ones, have been 
the prioritization of research over teaching (Scott, 2001). 


Institutional change happens when influential organizations in the field adopt and discuss new 
institutional logics or reframe existing logics. New or emergent logics are best identified through 
the language and discourse of groups because they have not become instantiated into institu- 
tional practices. Institutional logics gain greater legitimacy when multiple stakeholders adopt 
the logic (Bastedo, 2009). One of the critical roles for AAU is to have multiple field players - 
groups involved with STEM reform — adopt similar language or logics. 


Institutional logics are typically studied at four levels that help understand their dominance in 
shaping the field: content, penetration, compatibility, and exclusiveness (Scott, 2008). Content 
refers to the actual beliefs and assumptions and how compelling they are, which affects how 
they will guide/shape behavior and gain legitimacy. Penetration refers to their vertical depth and 
horizontal breadth. An institutional logic has penetration if it resonates with most people through- 
out an organization (although not necessarily everyone) and is widely found across institutions 
within the field. Compatibility has to do with whether the logic is related to institutional arrange- 
ments. In other words, a particular logic needs to fit with other belief systems and activities that 
guide the organization. Finally, exclusivity refers to whether other logics exist and the extent to 
which the logic is contested within the field. Logics that /ack solid content, penetration, compat- 
ibility, and exclusiveness are likely to have min- 

imal dominance within the field (Scott, 2008). We document how AAU successfully 


established a contradiction among 
We examined the emergent institutional log- sot bine ¢ fivate chi 
ics in the AAU Initiative and the extent to ©X!StINg logics to ne Ivate change 
which the Initiative appeared to shape and worked to revise the current 


changes. To the extent possible we explored institutional logic defining prestige. 
the content, penetration, compatibility, and 

exclusiveness of those logics. In this chapter we document how AAU successfully established 

a contradiction among existing logics to motivate change and worked to revise the current 


institutional logic defining prestige. Establishing the AAU Initiative was an important way to 
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establish credibility in promoting logics designed to enhance the quality of undergraduate 
STEM teaching. Some respondents believed that further change in AAU’s culture, particularly 
by making teaching excellence a criterion for membership, was needed to reduce the exclu- 
siveness attached to the logic favoring research. We discuss this issue in more detail in Chapter 
13, including the challenges for AAU in attempting to change its membership criteria. 


We note that respondents typically used the words “framing” and “messaging” when referring 
to institutional logics. This chapter explains the importance of how the Initiative was framed and 
messaged to create motivation for change, garner ownership, and supplant existing values that 
guide day-to-day actions. 


Establishing Crisis or Contradiction. According to institutional theory, identifying a con- 
tradiction or crisis is a precursor to motivating institutional actors to change. The data indicated 
that AAU was very successful in establishing contradiction. First, AAU pushed the concept “great 
institutions cannot be poor at teaching.” For elite institutions, all with a superior reputation for 
research, to have a central function, teaching, known as or even suspected to be mediocre is a 
major contradiction. Second, AAU asked its member institutions the following question: How 
can research universities which pride themselves on research not use evidence to assess and 
inform their teaching? Both contradictions were compelling to all groups interviewed. 


Some stakeholders felt that identifying a sense of crisis could contribute to the STEM reform effort. 
The outsized failure rate of racial/ethnic minorities in STEM was cast by some respondents as a cri- 
sis. They recommended that AAU emphasize the outsized gains by racial/ethnic minorities from 
active learning strategies, in part as a response to the crisis with these student populations. 
Another set of interviewees noted how the Initiative could be improved if it was connected to 
larger existing issues that are the priorities for presidents. 


In any case, respondents widely supported AAU’s success in identifying poor undergraduate 
instruction as a contradiction among its member institutions. 


Recreated Logic - AAU Elevating Teaching to the Level of Research. AAU developed 
framing for a new institutional logic® based on some version of the phrase: “AAU institutions will 
be as excellent in teaching as they are in research.” This broad institutional logic resonated strongly 
with most groups of interviewees. In a recent Inside Higher Education article (Oct 2, 2017) and in 
the introduction to the AAU report on the Initiative, AAU President Mary Sue Coleman described 
making excellence in teaching a priority “the new normal” for member institutions. 


When it came to compatibility and exclusiveness, various interviewees raised concerns about 
feasibility of or actual commitment to this logic. As one collaborator noted: “! think it's admira- 
ble what they’re trying to do in terms of focusing on excellence in teaching, but | have my res- 
ervations about whether AAU presidents will really get behind this and | really doubt there will 
be changes to the reward system. So, there’s lots of issues that | think will get in the way of this 
commitment.” We collected dozens of statements from interviewees concerned about the com- 
patibility of the logic about excellence in teaching, most of it centered on skepticism about AAU’s 
willingness to modify membership criteria. 


The AAU STEM Initiative focused on redesigning undergraduate STEM introductory classrooms. 
The rationale for this choice is that most reform efforts focus on co-curricular activities when it 


8 Institutional theory would this change a shift or reframing of a current logic, i.e., expanding the definition of prestige to include 
teaching and research. 
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relates to student success in STEM. Very little has been done to alter classroom teaching practices 
in introductory STEM courses, even though their enrollments at all institutions are among the larg- 
est. Undergraduate education at research universities has long been criticized. Many respondents 
mentioned Ernest Boyer’s critique of the teaching at the research university in his book College. 
About 75 percent of respondents agreed with this framing of the problem. 


Although in general agreement with the emphasis on teaching excellence, about 25 percent of 
respondents found the focus on introductory courses in STEM limiting. Most common was the 
objection to the lack of focus on diversity in the framing of the Initiative. One faculty member 
describes why diversity is an important issue in the STEM teaching context: “I still can’t understand 
why we are not focused on the success of students of color and women in STEM when this has so 
clearly been a problem for a long time and is getting a great deal of national attention.” AAU’s 
response emphasized that supporting undergraduate reforms would also benefit students from 
diverse backgrounds. This perspective seemed inadequate to several respondents who argued for 
a more targeted strategy for students of color. 


The other primary issue was whether the Initiative was a one-time reform, or if it was meant to be 
a continuous ongoing innovation. Respondents understood that AAU’s intent was for the reforms 
to be long-lasting, including the Framework’s emphasis on cultural change. The question is whether 
or not project sites viewed the Initiative in the same manner: “as institutions going through a one- 
time redesign of courses and then the AAU teaching issue will be done and addressed.” The pri- 
mary concern here is that a one-time project is less likely to impact the local culture. As one admin- 
istrator described: “| think it’s unfortunate — the language that they’ve [AAU] used. Reform is the 
word that’s often brought up, and | think if we talked about this as teaching innovation and some- 
thing that’s ongoing and that institutions need to continuously learn from their data about how to 
make a strong teaching and learning environment, that would be much more aligned with what | 
think we're really trying to achieve.” In our assessment many efforts by AAU are aligned with insti- 
tutionalization and long-lasting reform. For some participants, perhaps reframing the Initiative as 
an “ongoing innovation” might better support the notion of cultural change. 


Additional challenges included perceptions that graduate education would have been a better focus, 
that active learning pedagogy was too narrow a focus, that the focus should be on student learn- 
ing over teaching, and that all disciplines, not only STEM, should be included. 


All of the interviewees were attentive to the role of language in shaping actions of people on their 
campuses. We recommend that in future initiatives AAU and other national organizations ensure 
that to the extent possible the language represents a consensus of meaning among stakeholders. 


Framing the Approach to Change. In terms of the framing of the change process, interview 
data indicated strong consensus about targeting work at the institutional, departmental, and sys- 
tems levels; among both faculty and administrators; and, adopting a cultural approach. All groups 
interviewed described similar language for the process of change. Stakeholders agreed that the 
focus on institutional or cultural change was the right direction. As an example, one collaborator 
stated: “As | understand it, they [AAU] are taking an institutional approach. Past efforts have 
focused a lot on changing individual courses and that just hasn’t had much impact. So, | think they 
really, we are really on to something.” There was also consensus on the importance of focusing on 
departmental changes. As one faculty member stated: “The departments are the lifeblood of 
research university; this is where curriculums are created, where promotion and tenure decisions 
are vetted, and where values are established. AAU made the right choice in focusing on the impor- 
tance of departments.” 
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All groups interviewed emphasized the importance of faculty leadership. Interviewees said that 
engaging faculty as leaders in the change process was critical to its success. Respondents also 
acknowledged that support from presidents and provosts also was needed for successful imple- 
mentation and scaling. According to one faculty member: “We've had faculty champions in the 
past but if they aren’t supported by changes in the classroom, professional development, and 
a sense of priority on campus among administrators then these initiatives get very little 
traction.” 


There was also substantial support for aligning changes in instructional practices with the cul- 
ture and context of research universities. As an example, one administrator stated: “If we were 
to march out a set of ideas from other projects at a community college or liberal arts college — 
in fact this kind of thing has been done before — they just don’t get picked up by the research 
universities. It’s not as if there aren't other teaching and learning efforts going on in STEM, but 
this one is led by peer institutions.” Lastly, collaborators and AAU leadership viewed positively 
the emphasis on making changes systemically — working with the project sites, across the AAU 
Network, and partnering with other external groups including the National Science Foundation 
focused on improving teaching in undergraduate STEM courses. Although individuals at project 
sites generally lacked experience with efforts to partner with external groups, there was gen- 
eral consensus on the importance of working with these groups to effect local change. 


The AAU Framework codified many aspects of the change process. Although we found signif- 
icant agreement about the overall framing of the change process among stakeholders, there 
was less understanding among faculty and administrators of how to translate the Framework 
into change strategies. The faculty, in particular, found the Framework difficult to comprehend. 
They also had difficulty conceptualizing all components of the Framework simultaneously. 


Although there was consensus around the framing of the change process, there was often a 
lack of understanding about how to implement this direction.? Faculty members might describe 
the importance of institutional or cultural change but did not fully understand what that meant. 


Respondents found it difficult to implement a a 5 : 
change process they did not understand well. The positionality of different 


Similarly, collaborators, project site leaders, jndividuals within the system 


and AAU network points of contact reiterated appeared to shape their abilit 
the importance of top-down leadership sup- PP p y 


port but were less certain how to operation- to understand the approach or 
alize such support. The positionality of differ- 

ent individuals within the system appeared to eiieveay 1 change. 

shape their ability to understand the approach or strategy to change. Although interviewees 
typically did not describe the Framework as instrumental in supporting change on their cam- 
puses, their perspective likely was a result of their lack of understanding of the implications of 
the Framework for project implementation. We make recommendations for how to deal with 
this disconnect in Chapter 11. 


Disseminating and Supporting Recreated Logics. AAU used a variety of strategies to dis- 
seminate and embed the recreated logics. The primary strategy was the dissemination of the 
logics through various networks that AAU had created as part of the Initiative. The points of 
contact of the AAU STEM Network served to disseminate the new logics as well. Networks were 
the most scalable avenue for disseminating the logic on an on-going basis. 


9 We explore the intersection of perspectives about change by different groups in Chapter 11. 
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AAU also used a variety of communication strategies. The AAU president described the focus 
of the Initiative at each meeting of the AAU presidents and provosts. The new logic was also 
listed on the AAU website, noted in listserv messages to the campus points of contact and proj- 
ect site leaders, and included in newsletters and other outreach. The Framework document was 
a frame of reference at project meetings. Its visualization served as an avenue for disseminat- 
ing the new logic and a reference point for the logic in the change process. 


Networks and communication were the primary dissemination mechanisms, complemented by 
other activities in the Initiative. AAU also structured all of its meetings and events around the 
logic of the Framework. For example, breakout sessions at all of their events were organized 
around pedagogy, scaffolding, and cultural change. The project sites were a means for dissem- 
inating these new logics into practice. Site visits and grants that supported campuses to visit 
one another also disseminated logics. Several interviewees stated that “conversations about 
teaching excellence were occurring now on their campuses after an AAU staff visit” and that 
“having faculty or administrators involved with AAU STEM Initiative come to their campus” 
helped them think about ways to improve teaching approaches in STEM. 


Interviewees talked about mechanisms to support this new norm for improved undergraduate 
teaching—in institutional theory the recreated logic—over time. Respondents found that hav- 
ing peer institutions involved in similar change processes gave them some political protection 
with their faculties and administrators. As one faculty member stated: “I don’t think our cam- 
pus would move in this direction unless there were a whole set of campuses doing this work 
together. There is safety in numbers.” Respondents also indicated the potential usefulness of 
common standards and assessment tools across peer institutions, although agreement on such 
metrics is not widespread at this time. Local norms and traditions for measuring teaching also 
can work against the use of common standards. 


Conclusion. One of the most difficult parts of the change process is developing compelling 
logics that can motivate long-term cultural change and shift entire symbolic and language sub- 
systems. The study of the AAU STEM Initiative provides a helpful articulation of the way logics 
can be developed, disseminated, and supported. Although logics have been created and dis- 
seminated, little support exists for them at the current time. The compelling logic that they cre- 
ated provides the motivation and buy-in to begin the change process. The chapters on influ- 
ence, networks and learning articulate ways logics are being disseminated and supported. 
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Creating and Leveraging 
Networks for Change 


Evidence strongly suggests that AAU was effective in creating and leveraging networks, a fun- 
damental role in supporting undergraduate STEM reform. Consistent with the literature on 
organizational change, these peer relationships and institutional/interpersonal networks are 
crucial factors in changing ideas and practices (Eckel and Kezar, 2003; Fairweather, 2008; Rogers, 
2003; Valente, 1995). As one respondent indicated: “If you think about it, there’s a massive 
amount of networks set up around research. Disciplinary societies support and focus on research. 
There’s a gigantic research infrastructure within every AAU institution. What we need are net- 
works, prestigious ones, around teaching. That would change everything.” 


In this chapter, we review how AAU serves as a “network of networks.” We first describe the 
types of networks leveraged to implement change. Next, we describe effective practices used 
by AAU to leverage the networks for change. Lastly, we highlight a few missed opportunities 
for leveraging networks. 


Types of Networks. Interviewees described various networks potentially relevant to improving 
undergraduate STEM education at AAU institutions. Part of the effectiveness of the AAU 
Initiative was its ability to navigate and work with so many networks varying in composition, 
size, and levels of work. For example, AAU worked with constituent groups including individual 
AAU campuses, STEM reformers, national groups such as POD and CRUSE, and funding agen- 
cies. Figure 2 visualizes these networks: 
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AAU Project Sites: The AAU engaged the eight project sites regularly through meetings 
and other forms of communication (e.g., teleconferences, webinars, listservs), essentially form- 
ing a network of project sites. As one consequence, over time the project site teams became 
familiar with one another's work. A participant described the project site network as follows: 
“so the various campuses that receive the money they don’t just go off and do their own 
work. AAU is bringing them together regularly. This is really important that the sites interact 
regularly and share information. This means that there is much greater likelihood for a big- 
ger impact across the campuses, they're not just isolated sites.” 
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AAU STEM Network: All 62 AAU member institutions were invited to participate in the AAU 
STEM Network. Invitations were directed to the point of contact on each campus. During the 
evolution of the Initiative the Network meeting became an annual event. Many interviewees 
agreed with AAU'’s decision to invite all member institutions to the Network instead of limit- 
ing it to the 31 institutions submitting a proposal to become a project site. As one example: 
“Our campus already had a lot of energy around innovation in teaching, so this allows us an 
opportunity to connect with others and get ideas. We have a really active biology department, 
but our math department is interested in getting involved but doesn’t have a lot of direction. 
So, we're able to get ideas for [the] math department from our involvement.” 


AAU STEM Community: During the Initiative an AAU STEM “community” emerged from 
various networks — project sites, AAU STEM Network, and other external stakeholders. Several 
individuals echoed the sentiment that: “One of the best outcomes of the initiative was a 
larger community of AAU peers that care about STEM innovation.” Interviewees described 
this community as larger than the AAU STEM Network — a community encompassing a diverse 
set of individuals deeply committed to STEM reform at research universities. One faculty mem- 
ber described this community as follows: “The real value for me out of the initiative has been 
the development of the community of peers dedicated to STEM innovation. There are many 
people now | can call on and | just didn’t have that in the past.” This community was espe- 
cially important for individuals working on campuses without significant ongoing STEM edu- 
cational reforms. This community developed organically out of the other networks. Many 
respondents believed the AAU STEM community was among the most important achieve- 
ments of the Initiative. In the future this community seems well positioned to support insti- 
tutional and cultural change. 


Subgroups: Subgroups within AAU institutions also formed informal networks. These sub- 
groups included directors of Centers for Teaching and Learning, faculty in particular disci- 
plines such as biology or physics, and department chairs. AAU convened a special meeting 
for the interested to take advantage of the momentum in the Initiative. One faculty member 
described the value of these subgroups: “| now talk to biology professors across a number 
of AAU research universities. We're exchanging ideas about curricular reforms and assess- 
ment, and mostly we're just supporting each other.” The evolution of subgroups was an 
important outgrowth of the Initiative, one because of its organic nature that will likely be 
sustained as the Initiative ends. 


Additionally, AAU also involved external networks. Interviewees found substantial value in net- 
working with important external organizations such as the National Science Foundation and the 
National Academy of Sciences. According to one respondent: 


“The AAU project connects to people that are outside. The ability to connect people 
from different realms, so let me put it that way. The AAU project brought us to talk to 
the provost of all the AAU institutions. They have brought us to interact with the Cottrell 
scholars. They have connected us to HHMI, HHMI scholars, HHMI professors. The other 
projects don’t do that. The other projects are more insular to themselves. | think the 
AAU in this work has been more broadly connected and they have connected the 
Bayview Alliance and they have connected just across a lot of different networks. 

You start to see who overlaps and you start to see some new things. That’s been 
useful for our own development here on our campus. That's been useful.” 
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In this next section, we elaborate on these valuable external connections. 


Coalition: One of the major Washington D.C.-based groups focused on improving under- 
graduate education is CRUSE (noted in Chapter 4). AAU joined and supported CRUSE. One 
of the AAU STEM Initiative senior scholars, Linda Slakey, was the convener and facilitator of 
this group. CRUSE has worked to align and advance work across multiple national projects 
aimed at improving the effectiveness of undergraduate STEM education. 


Networking with external groups: AAU also connected to existing networks to expand 
the expertise available to the Initiative. Examples of this type of external networking include 
connecting with POD — a national organization dedicated to improving teaching and learn- 
ing in higher education. AAU also linked with other STEM reform projects like the BVA, CIRTL, 
APLU’s NSEC project, and AAC&U's PKAL and TIDES projects. An administrator found these 
linkages important: “There are no other opportunities for me to be in a room with national 
higher education organizations, experts in STEM teaching like the Bayview Alliance or lobby- 
ists from the AAU. All of these different perspectives have been helpful in thinking about 
innovation on our own campus — how to get people’s attention, how to frame things.” 


The AAU’s STEM Initiative Advisory Committee also connected the Initiative with important 
external groups. As one interviewee describes: “The advisory board is a real asset to the over- 
all initiative. We can call on them to invite speakers, to connect us with other STEM reform 
projects, to let us know about new publications coming out or policies that will affect our 
work, to call up and put pressure on leaders on campuses’ project sites, to tell us about exam- 
ples of good work happening on an AAU campus.” 


AAU was less successful in engaging disciplinary societies in part because it had “little history 
in working with these groups and little visibility among them.” As a prestigious organization, 
interviewees felt that AAU could link with disciplinary societies in the future to expand the 
reach of the Initiative. 


Functions of Networks that Facilitate Change. Different types of networks within the 
Initiative serve different purposes. For example, interviewees participating in the AAU STEM 
Network and project site networks particularly benefited from their focus on knowledge, learn- 
ing, and emotional support. Below we describe important network functions, including influ- 
ence; knowledge, learning and brainstorming; funding and resources; dissemination; emotional 
support and encouragement of risk taking. 


Influence: The most frequent benefit cited by respondents was their role in influencing others. 
Many interviewees described the influence of having prominent speakers and participants in net- 
working events. For example: “AAU is able to get the most prestigious and well-known speak- 
ers. Because of that, they have had really high attendance at meetings. | was surprised by that, 
especially given the focus of the initiative.” AAU can attract major speakers such as Carl Wieman 
or organizations such as the Howard Hughes Medical Institute in part because of AAU’s connec 
tions to other STEM reform activities and work, which implies the visibility of the AAU Initiative. 


Knowledge and information: |nterviewees also spoke of the knowledge they gained from 
participating in networks — such as models of good teaching, ways to evaluate teaching, and 
data analytics. A faculty member commented: “To have a group of people that you can call 
on if you have a challenge and that they can help you think it through. That’s what this net- 
work is — a group that is a source of information.” Networks were crucial to promoting the 
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use of evidence-based teaching methods and data analytics in large part because of AAU 
involvement. Interviewees acknowledged that the same information disseminated through less 
prestigious networks would have had less impact on their campuses. Ultimately the prestige 
of AAU was as important as the type of information disseminated by the various networks. 
Information sharing was important to stimulate STEM reforms on individual campuses. A cru- 
cial element was sharing information with peer institutions: “It’s true that having connections 
to other schools is an incubator for ideas, and it’s [change] more likely to happen, the more 
| hear what the other AAU schools are doing than if we weren't.” 


Brainstorming and learning: Respondents found networking useful to rethink their teach- 
ing and brainstorm about new practices. Equally important was finding ways to prioritize 
teaching, especially through modifying reward structures. Consider the following statement 
as an example of these effects: “We have this aha moment, when several of us [department 
chairs], we're talking about how to get more buy in for student learning outcomes and assess- 
ment. Sure, you can hear about different approaches to doing this work, but we're really 
struggling with getting buy in among the faculty to use them — the insights that we came to 
and wrestling with this problem together was valuable.” Although many respondents could 
see the potential of AAU-related networks to enhance their own learning, this potential was 
not always realized. Finding ways to make sure that the potential for learning through net- 
works is more effectively realized is a crucial endeavor for the AAU moving forward. 


Emotional support for change agents: Faculty and administrators found attempting to 
prioritize or at least enhance the value of teaching in a research university time consuming 
and emotionally draining. Many interviewees found the AAU-related networks important 
sources of emotional support. As one example: “I really appreciate coming to the event and 
talking to people because then | don’t feel so alone in this work and it really provides moti- 
vation and energy to continue to do this work on my campus.” Another respondent stated: 
“You can always be in a bubble a lot, right, and you think, sometimes, that the unique chal- 
lenges that you’re having in your institutions are yours alone. It’s sometimes good to hear 
that your peer institutions are having the same challenges. What | took away from that was 
(a) we're not alone, and (b) there are some 

schools doing some stuff that we just “| definitely think that there 


needed to keep an eye on.” Emotional sup- . . 
port is a less tangible but important bene- has been this safety in numbers 


fit of networks, one that AAU should con- mentality, that knowing there are 
tinue to emphasize in its future work. many different AAU institutions 


Safety in numbers and reducing innova- involved in improving teaching, 
tion risk: Having peer institutions involved that has made a difference for 


in the improvement of undergraduate STEM . ATE 
education made it safer for individual faculty faculty being willing to try and 


members and administrators to support @dministrators to giving us a 
change on their own campuses. As one sense of priority, of support.” 
interviewee noted: "I definitely think that 

there has been this safety in numbers mentality, that knowing there are many different AAU 
institutions involved in improving teaching, that has made a difference for faculty being willing 
to try and administrators to giving us a sense of priority, of support.” 


Resources and funding: Many respondents cited the importance of AAU's links with orga- 
nizations that fund STEM education such as The Helmsley Charitable Trust, NSF, HHMI, 
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Research Corporation for Science Advancement (RCSA), the Alfred P. Sloan Foundation and 
the W. M. Keck Foundation. As one administrator noted: “| can get faculty and administra- 
tors to go to meetings when | can tell them there will be funders there. It’s also helpful to 
hear how funders are considering changes in direction, so | find it interesting as well. So, 
the funders provide real motivation for participation.” 


Dissemination: As described in the previous chapter, AAU employed networks to help dis- 
seminate best practices and shape values and norms about teaching on its member campuses 
and in the broader higher education community. For example, the connection with the UC 
System and BVA was an effort to further disseminate logics and best practices of the Initiative. 


The AAU Initiative also used on-campus networks to help scaling by encouraging links 
between teaching innovators in STEM and non-STEM departments. Consider the following 
example from one faculty member: “I have talked with people from English and philosophy. 
| actually was giving a workshop and talk yesterday to a group of faculty. My co-faculty work- 
shop presenter was from the philosophy department. We were talking about many of the 
things that we have in common, how we might use some of the same design and online tools 
to help students write and draw, and they need to write and draw in philosophy, just as much 
as you are doing chemistry.” One administrator summarized the cumulative effect of net- 
works in the AAU Initiative: “Now, | can say — | know some people across the country who 
are working on such things. | can pay attention to what they are doing. | know if | see an 
article in Science or a position piece in some other magazine, | recognize the name from one 
of those conferences. | read that much more closely. | know | can trust it. | like that.”"° 


Effective Network Practices. Interviewees described several key factors in the effective use 
of networks in STEM education reform. These included identifying peer networks, providing 
opportunities for networking, and identifying network subgroups. 


Identifying peer networks: |nterviewees strongly emphasized the importance of engaging 
peers in change efforts. By peers, interviewees meant both peer institutions and peer profes- 
sionals. “Peer professionals” referred primarily to the faculty and administrators in other AAU 
institutions, especially those from the same discipline. As one administrator described: “So 
why are we able to create changes on our campus? A main factor has been being able to say 
that we're part of this broader AAU STEM Network and our peers are making these changes 
and if we don’t want to get left behind we need to start doing this work as well. That really 
motivated a lot of changes in terms of getting resources, space, and even some faculty buy 
in.” As another stated: “Il am the only one in an administrative role my campus dedicated to 
improving teaching. So, meeting with others who have a similar role on another campus has 
been so helpful to share advice and feel less isolated and provide emotional support.” 


Providing opportunities for networking at meetings and through activities: Allowing 
time for participants to interact in unstructured formats during AAU STEM Network meetings 
increased the impact of networking. As on interviewee stated: “AAU creates great networking 
potential at these meetings. First, they invite the right people, influential people. And there’s time 
in between the talks for participants just to mingle. I've followed up with a few people that | met 
at meetings.” AAU staff also used the site visits to encourage networking by putting local faculty 
and administrators in touch with relevant faculty, staff, and administrators at other locations. 


10 It should be noted interviewees did not state that AAU helped support the development of on-campus networks, but this might 
be an area for future consideration. 
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Identifying subgroups: Breaking into subgroups — by discipline, role, or area of work — was a 
successful tactic at AAU STEM Network meetings. According to respondents, the use of these 
subgroups was especially important because the context of changing teaching practices typically 
is most strongly influenced by the academic unit and by peers: “It hasn’t happened very often, 
but when we've broken up into disciplinary or role groups we've been able to tackle some meaty 
problems that lots of us are facing on campus, but individually are having troubles addressing.” 


Technology-based networking: |n addition to face-to-face strategies, AAU used various 
technologies in networking including electronic platforms, listservs and emails, websites, and 
curated resources. Interviewees found these strategies less successful than in-person commu- 
nication. One faculty member described the difficulty in using social media to network: 


“I'd say all of them [organizations] struggle with having ongoing communication happen. 
All of them struggle with ‘How do you keep people connected and working together 
when they are not meeting? If you are going to meet twice a year, what happens the 
rest of the year?’ All of them struggle with having effective means of doing that. The 
AAU has tried. First, there was the products on Purdue, and now they are using Trellis 
and it’s kinda hard to have any of those things work. | mean, | haven't seen anybody do 
that well and we struggle with the same issue with some of our projects, how to really 
keep a community going virtually is very difficult.” 


Future Directions for Networking. Interviewees also identified needs for improvement in 
networking activities including identifying the right network members; empowering the AAU 
STEM community to lead; creating network leadership; tapping existing president, provost, and 
dean networks; broadening on-campus teams; networking department chairs"; and altering 
the network to expand its reach. 


Identifying the right points of contact/network members: The point of contact at each 
AAU campus was an important link to help AAU establish and operate the AAU STEM 
Network. In many cases the individual appointed had the necessary contacts and sway to 
operate effectively and encourage the use of evidence-based teaching in STEM. In a few 
cases, the point of contact was not sufficiently well positioned or experienced to carry out 
networking responsibilities and foster change on campus. When the point of contact was 
unable to further the AAU Initiative on campus, in a few cases the university changed the 
point of contact which increased the viability of STEM reforms on their campuses. As one 
point of contact stated: “So I'm the new point of contact on the campus. The last person, 
she stepped down, as she had no background in STEM education. She just didn’t know who 
to communicate with on-campus and she wasn’t interested in attending the events of the 
AAU Initiative, but she was in the role for three years. So now, I’m just attending my first 
event and | started to communicate across campus and there’s a lot of excitement. So, when 
| go back to campus | am going to be giving a talk about what | heard at the meeting. But 
before, nothing was communicated.” Identifying a potentially effective point of contact on 
each campus is an important part of the success of the AAU Initiative. 


Empowering the AAU STEM community to lead: Continuous support of some form is 
needed to sustain AAU-related networks for undergraduate STEM reform. As one faculty 
member described: “You have this passionate group of individuals committed to change, and 


11 Based on the success of the first department chairs workshop, AAU will convene AAU STEM Department Chairs biannually 
starting in 2018. 
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now they're connected across all these different campuses, all these different influential cam- 
puses. But | don’t hear anyone fostering this group. A network needs resources, a platform 
or some infrastructure, and it can also help if there is ordaining of its legitimacy by AAU. So, 
it exists but it’s not supported at this point or truly empowered to play the role it can.” The 
future success of the AAU Initiative depends in part on AAU’s ability to find ways to continue 
supporting the networks working to improve undergraduate STEM education. 


Network leadership: To continue to be successful the complex “network of networks” 
requires continuity of leadership of both individuals and groups. As one administrator 
described: “With other networks that I’m a part of, there’s a real passionate leader that 
embodies the community. I’m not sure who that would be with the AAU STEM Initiative or 
AAU STEM Network that is emerging. And | worry because the leader provides the emotional 
energy and motivation for people to continue as well as a sense of direction.” Most individ- 
uals interviewed felt that the AAU STEM community was the most efficacious network — the 
network of champions who want to improve STEM teaching and learning at AAU institutions. 
AAU could help in this effort by recommending individuals or groups for leadership roles in 
the network or by helping create an executive committee to oversee these activities. 


Utilizing existing president, provost, and dean networks: |nterviewees differed in their 
views about how well AAU used its internal connections — especially those with presidents 
and provosts — to promote local change. Documents suggested that AAU did have signifi- 
cant contact with these important constituencies about the Initiative. Yet many campus par- 
ticipants did not know about these efforts. Consider the following example: “When | think 
of AAU, | don’t think of STEM reform. | think of presidents. And from what | can tell, there 
isn't a great deal of knowledge among presidents about this Initiative. This seems an untapped 
resource. Why are presidents and provosts not talking about improving teaching.” Despite 
the sentiments expressed by this (and other) respondents, site visit interviews showed that 
presidents and provosts typically were aware of the Initiative. Many of them helped spread 
the word and provide resources. Yet respondents were looking for more from campus lead- 
ers especially in the form of public support for quality teaching in faculty rewards. 


It may be that some individuals look more to outside groups such as AAU than to internal 
dynamics when attempting to make teaching more of a priority on their own campus. From 
this perspective the external push for change is given as much or greater weight than with- 
in-campus dynamics. Consider the following example. One dean who attended a meeting of 
AAU deans” said he was supporting the promotion of a faculty member based primarily on 
good teaching. None of his colleagues said they would have supported such a case on their 
campuses. The dean’s view was that AAU’s connection with presidents and provosts did not 
result in sufficient encouragement to deans to promote instructional improvement. 


These examples show the complexity and tension within the broader movement to improve 
undergraduate teaching at research universities. On one side—reflected in the above exam- 
ples—is the need for direct external peer pressure to assist local campuses in leveraging 
peer actions to make needed changes. An alternative perspective views changing attitudes 
about the importance of teaching as primarily a function of local cultures (department, col- 
lege, institution). From this latter perspective the AAU is an indirect source of influence in 
the change of local cultures. The literature clearly indicates that outside pressure alone is 


12 During the Initiative AAU held its first-ever meeting for deans from AAU campuses. AAU plans to continue such meetings. 
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not determinative in enacting local change. In our literature review, as reflected in the AAU 
Framework, both internal and external networking are crucial to organizational change. 
Among AAU’s most important contributions is bringing external peer pressures to bear on 
local reforms. AAU is still learning how best to leverage these networks in the service of 
local improvements. 


One problem is the lack of communication between AAU and individual faculty and admin- 
istrators about its regular convenings of presidents and provosts. Our review of documents 
and observations found that the Initiative was a regular topic of discussion at AAU spring and 
fall meetings of the presidents and chancellors. Yet, because AAU meetings are closed to 
outsiders, these proceedings typically were not disseminated broadly. It may be presidents 
and chancellors of AAU members find it easier to talk about improving teaching than to 
increase its value in promotion and tenure decisions. 


In the future, it is critical that there be a way to clearly communicate about the importance 
of undergraduate teaching by the presidents and provosts, perhaps through public state- 
ments. Additionally, venues where presidents and chancellors can engage directly with 
department chairs and faculty members to discuss the value of teaching on the campus is 
symbolically significant. As one respondent stated: 


“| think one of the problems is that we don't really fully understand or talk about what 
senior leadership support means. | think that we need to call out and activate the kinds 
of support we need from senior leaders to make these changes real. It might be a list of 
suggested actions for presidents and provosts, like establish a task force to examine 
better ways to measure teaching on your campus; set aside a certain amount of the 
budget for infrastructure for better teaching; hold departments accountable for quality 
teaching. So, we need to actually help administrators understand what they can or 
should be doing quite specifically — that, to my understanding, has not happened yet.” 


AAU might also work to provide additional “I think one of the problems is that 


concrete examples in case studies and dis- wea don’t really fully understand or 
seminate potential strategies to engage 


senior leadership in the change process on talk about what senior leadership 
campuses and across the AAU network. support means. I think that we need 


One example of such work was the case . . 
study publication developed by AAU for to call out and activate the kinds 


deans. More of this type of work will be Of Support we need from senior 
important to continued cultural change. leaders to make these changes real.” 


Broadening on-campus teams into institution-wide networks: |nterview data showed 
that most project teams were not connected with a larger campus network or community. 
Most teams did not even consider how they might create an on-campus network or expand 
and scale their efforts. Interviewees did not feel that AAU encouraged them to form a local 
network or connect with other STEM reform projects. Only one project site team described 
how they had expanded into (what could be considered) a network that connected faculty, 
administrators, and academic units across campus in an effort to improve teaching. This net- 
work was a local effort: 
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“We have the CIRCLE fellows. This is a two-year fellowship that was aimed at helping 
faculty implement active learning and creating a network. They get paid and they 

work with us for two years, which allows them to implement some sort of intervention 
the first year with our help. We get to evaluate it and then help them refine it to do 

it the second year. What we did was we picked the introductory courses. We picked the 
courses that have co-teachers. We worked with biology, chemistry, physics we're now 
working with, and chemical engineering. Over time this is expanding and 

connecting more faculty.” 


Observational data showed more signs of emergent on-campus networks. Some project sites 
formed faculty learning communities to create a type of network of those interested in 
improving STEM instruction. At another site, the AAU project was housed in a center coor- 
dinating various STEM initiatives on campus. 


AAU recognized the need to encourage greater development of on-campus networks in STEM 
reform. AAU received a grant in 2017 from the Northrop Grumman Foundation to help cam- 
puses link various STEM improvement projects, and to create greater synergy across projects. 
In addition AAU received an NSF IUSE grant in 2016 to study the within-campus dynamics of 
implementing and scaling STEM educational reforms. 


Networking department chairs: Project participants agreed with AAU about the centrality 
of departments in improving undergraduate STEM teaching. Some suggested that AAU move 
beyond its nascent efforts to convene department chairs to establish a formal network of chairs 
with a focus on STEM reforms. Although supporting the concept, others were not sure that 
AAU was best positioned to sponsor such an effort. In either case, finding a way to create a 
network of department chairs seems potentially an important part of future STEM reforms. 


Networking too focused on the core: A few individuals suggested finding a longer-term 
solution to the management of AAU-related networks. As one example: “The initiative builds 
networks, but they are all dependent on the AAU staff and initiative to connect. There needs 
to be more work to build up the connections that exist outside the core so that the network 
is more distributed and therefore more sustainable over time, and also scaling and disseminat- 
ing information more.” AAU leaders recognized that individuals in member institutions even- 
tually needed to take greater ownership for managing AAU-related networks. AAU took a 
step in this direction by asking several project sites to host future meetings of the AAU STEM 
Network. Balance is key here: AAU must maintain sufficient control for the Initiative to suc 
ceed while at the same time moving to distribute leadership across network participants. 


Conclusion. Networking was central to scaling innovation in STEM at AAU campuses and 
beyond. Networking builds on AAU’s historic strengths and capabilities, which made AAU well 
positioned to support networking. The AAU-related networks served many functions to sup- 
port the scaling of innovation. Recommendations for enhancing this already strong networking 
effort included encouraging the development of on-campus networks, identifying the right indi- 
viduals to participate in the network, and better leveraging the existing senior leadership net- 
works in local change efforts. 
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Information Sharing and Learning 


This chapter examines how networks facilitated information sharing and encouraged the use of 
best practices in undergraduate teaching. As the chart below illustrates, information sharing is 
an important but not sufficient contributor to learning and adoption of change. Nor is learning 
the only route to adoption of change. As we examine in this chapter, in this study we found 
more evidence of information sharing than learning. 


Learning is a complex phenomenon that includes unearthing underlying assumptions, challeng- 
ing existing assumptions, data and information systems and sharing, turning information into 
knowledge, inquiry processes, and brainstorming and group processes. Most of what interview- 
ees described was information sharing and, in a few instances, brainstorming, which are neces- 
sary precursors of learning and consequently adoption of innovations. Yet learning is not the only 
way to achieve adoption. In this study, adoption seems to occur more often as a result of influ- 
ence and new norms than as a result of individual and institutional learning. Some interviewees 
thought the lack of learning was a function of timing; learning would become more evident over 
time. Evidence also suggests that influence can facilitate learning (see Chapters 6 and 10). 


Most interviewees did not believe that learning was a principal component of the AAU Initiative 
strategy nor was it necessary. They indicated limited evidence of learning (or adoption vis-a-vis 
learning) as a result of the Initiative. Influence and networking were much more influential strat- 
egies than learning in the Initiative. Interviewees were less confident in AAU’s ability to promote 
educational change by learning processes than by influence and networking. 


Table 2: Definitions 


Information sharing 


Involves the distribution 
or dissemination of 
information, ideas 

and practices aimed at 
learning and adopting a 
new practice or policy 


Learning 


Involves a deeper en- 
gagement with infor- 
mation, comparing it to 
underlying assumptions, 
challenging existing 
assumptions, turning 
information into knowl- 
edge, inquiry processes, 
and brainstorming and 
group processes 


Adoption thru learning 


Involves implementation 
of an idea/practice by 
considering barriers, 
obstacles, facilitators 
and more thoroughly 
brainstorming how to 
make the adoption 
successful 


In this chapter, we review examples of adoption that seemed the result of learning, how AAU 
facilitated learning, and challenges to AAU’s use of learning as an adoption strategy. Most of 
the learning described by interviewees related to cross-campus experiences (e.g., hearing about 
a practice on another campus and working to understand it) rather than to internal learning 
processes where faculty members on a campus help their colleagues understand the use of inno- 
vative curricula or instructional practices. 
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Evidence of Adoption of New Practices through Learning. Respondents spoke less about 
specific learning experiences affecting STEM instructional practices and more about a general 
awareness or shift in perspective: “It’s more a general awareness than specific project initiatives 
that we've been trying to emulate. So yes, | can’t point to specific practices we've adopted or 
learning that occurred, but yet | feel some kind of learning has been gained.” These data may 
be limited because interviewees were not always aware of instructional improvements on their 
campuses so could not relate specific learning experiences to them. 


Also, interviewees sometimes could not distinguish between the effect of learning on STEM inno- 
vations from that of, for example, influence. In a few cases we found evidence of the effects of 
learning on STEM instructional reform. We describe the impacts of these learning processes next. 


Data analytics: The AAU Initiative successfully exposed participants to the use of data ana- 
lytics in portraying student retention and student learning outcomes. During the Initiative sev- 
eral campuses (both AAU Project Sites and members of the AAU STEM Network) adopted or 
considered adopting data analytics to improve STEM student success and to learn how instruc 
tional practices affected student learning 

and degree progression. As one point of The AAU helped campuses view 


contact described: “We're really excited qata on students and faculty as 
about the data analytics and not only how 


we collect a lot of information. We also have more than statistics to be reported; 
them [AAU leaders] come out to ourcam- they became sources of identifying 


pus to help people understand and adopt . . 
this practice.” The AAU helped campuses where instructional reforms were 


view data on students and faculty as more most needed and potentially 
than statistics to be reported; they became most impactful. 

sources of identifying where instructional 

reforms were most needed and potentially most impactful. One administrator describes this 
learning: “Our team went to several sessions — across different meetings — having to do with 
data on student learning, pathways, outcomes. All of that helped us to explore and think 
through our data systems and start to brainstorm ways we can improve.” 


Curriculum: Participating in the Initiative led some AAU institutions to modify or consider 
modifying STEM curricula, including adopting innovative STEM curricula from other AAU cam- 
puses. One administrator stated: “For example, our chemistry department is collaborating 
with a department from the University of Arizona in adopting an entirely new Intro Chem 
series. That was a direct result of participation in the initiative.” During the Initiative several 
respondents gained knowledge about assessing student learning, which helped them iden- 
tify shortcomings in their curricula and led them to consider alternative curricula and forms 
of assessment. The AAU survey data also stimulated participant discussions about the effects 
of curricula on student learning outcomes and retention, which led to proposed improve- 
ments in curricula and instructional practices and identification of needed improvement in 
infrastructure and policy. 


Discipline-based education researchers/faculty: Some AAU institutions hired or con- 
sidered hiring faculty members who conduct research on teaching within their discipline - 
discipline-based education researchers (DBERs) — as one way to improve teaching in STEM 
disciplines. As one administrator stated: “Our campus is now considering hiring DBER faculty 
— we currently do not have any. To have educators embedded within each department. We 
proposed this to the incoming president.” The challenge at AAU (and other) institutions is 
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where to place these faculty members — in STEM departments or in education? — and how 
to review their work for promotion and tenure. Particularly challenging are DBER faculty in 
STEM departments where the nature of research in the field is quite different than DBER 
research. Among the key questions were whom to use as external references and how to 
educate other faculty in the department to view DBER faculty members’ contributions to the 
department. Some respondents credited the AAU Initiative for helping them see the bene- 
fits of DBER-related hires and for assisting them in identifying the challenges of these types 
of hires in the promotion and tenure process. 


Active learning classrooms/facilities: Another element of scaling STEM innovations was 
the modification of classroom spaces to support active learning. The Initiative helped institu- 
tions considering such modifications learn from others who already had modified some class- 
rooms. As one administrator explained: “An area we're really interested in is better class- 
rooms, and the initiative has allowed me to learn about and visit some novel learning spaces.” 


Pedagogical practices: Some interviewees cited the importance of discussing or even 
observing active pedagogical practices prior to implementing them on their own campuses. 
One faculty member described observation tools that they considered adopting: “At Davis 
they have tools for evidence-based action, | think that’s it. | think that would really resonate 
on our campus. So, there are active discussions happening about that.” Another administra- 
tor noted the teaching practices inventory: “| think the one thing that comes to mind that 
we're thinking about adopting is Carl Wieman’s teaching practices inventory and trying to 
think about how do we incorporate that into what we do. Is that something that we man- 
date? Is it something we suggest that the faculty use as a tool for themselves? So that’s an 
example of how | hear something at a meeting and | try to work with others to incorporate 
and brainstorm the ways it would work with our own particular culture here.” The AAU 
Initiative annual meetings were a primary source of information about active pedagogical 
strategies for those considering adoption on their campuses. 


An obvious source for modeling active learning strategies is other departments or programs 
on the same campus. Here the evidence of institutional learning was mixed. Several campus 
teams reported some adoption by new departments and new colleagues within depart- 
ments. Others found spreading the use of active pedagogies beyond the initial adopters 
problematic. 


Learning that change can be institutional: Some respondents learned to consider ped- 
agogical change from an institutional as well as a classroom perspective. The Initiative in its 
Framework helped some participants consider how institutional context influenced what oth- 
erwise seemed to be departmental or program-level matters. As one example: “Something 
that came out of this initiative for me was an appreciation that good teaching — to make that 
happen — it will require institutional support and commitment.” 


Learning about the change process: Some individuals reported learning about the change 
process. As one faculty member noted: “Noah Finkelstein as an example. These are individ- 
uals who were very much in tune to organizational change theories that, in my mind, had 
always been under the purview of higher education. It makes the task of trying to approach 
social transformation much less daunting because there are these people out there doing 
that and helping you to understand how to do it. | think AAU does a good job highlighting 
where those best practices about change are happening.” 
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Although it was not always clear whether respondents had adopted pedagogical reforms, 
interviews indicated that they learned about adopting practices, becoming familiar with new 
practices/ideas, bringing them to their campuses, and brainstorming ways they might be 
implemented. 


Reflecting on minimal learning or adoption of new practices: Some respondents 
found the pace and spread of learning and adoption of new practices from peer institutions 
on their campus slow. As an example: “I think one thing that AAU would have hoped, and 
it hasn’t really happened so much, is that there would be more cross-university collabora- 
tions and dissemination of ideas. We've just been so busy just taking care of our own ini- 
tiative, we haven't really collaborated much or borrowed from other campuses.” Some inter- 
viewees attributed the lack of institutional learning or its spread to the short time frame of 
the grant. 


Other individuals disagreed. These respondents emphasized the importance of increased 
awareness by faculty members and administrators about the use of evidence-based teach- 
ing practices at other AAU campuses as the crucial first step. These respondents believed 
that such awareness is crucial to making active teaching and learning an expectation or 
norm. In their view, over time the new norm would lead to improved instructional practices. 
These interviewees focused more on AAU’s influence and norm setting than on monitoring 
the extent of reforms on campus at this point in time. 


Practices that Supported Learning. The initial RFP contained ideas about how to struc- 
ture the projects sites, including the use of data, working collectively in teams, and working 
across departments. These ideas ultimately were instrumental in supporting learning. AAU site 
visits and Network meetings also fostered participant learning. Some internal campus practices 
such as working on project teams also promoted learning about effective reform strategies. 


Request for proposal: The RFP required institutions to develop an institution-wide proj- 
ect, to use data, to work across multiple departments, to engage senior institutional lead- 
ership in some way, and to make use of a cross-campus team with faculty members, admin- 
istrators and staff. Several interviewees described the RFP as a tool for learning. As one 
respondent noted: “I just really think it’s the integration of the idea that we had to have cur- 
riculum development, faculty development, evaluation, and cultural change together. The 
way we had to write our grant, we had to integrate it all together. We had never worked 
like that before.” Respondents consistently described the RFP as carefully crafted and influ- 
ential in the approach they adopted to improve undergraduate STEM education. The RFP 
became a pivotal reference document for learning. 


Use and collection of data and metrics: The use of data and metrics also contributed 
to learning. Interviewees described as influential several data sources, including the data 
analytics work by Marco Molinaro at UC Davis, the survey data collected by AAU about 
teaching, and the emphasis by projects on student learning outcomes. One faculty member 
noted the importance of data in encouraging adoption of innovations: “Well, one of the 
things that really brings people out is that we have data, but it’s not just the data, its data 
collected as part of AAU. People who are using whatever sort of active learning, we have 
data about their classes compared to non-active learning, we bring them together every 
once in a while. They don’t have to set anything up, but they do come. We bring them 
together, and they talk about the data. And that has led to insights into the value of active 
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learning and more adoption.”'? An administrator described how data were used to support 
learning on their campus: “We could truly measure and see what was going on now, how 
are things changing and/or through a big data approach could we, once all this data got 
together and it was integrated, could we find areas that needed improvement. That’s been 
really exciting.” 


Working across departments: Respondents found two key components of the RFP — work- 
ing across departments and developing a team with representatives from across the campus 
— instrumental in fostering learning. Requiring departments to work together resulted in 
shared ideas about teaching, assessment, and evaluation, all topics not typically shared across 
departments. This arrangement helped spread the use of evidence-based pedagogies by 
exposing potential adopters to those with extensive expertise and experience in such strate- 
gies. A faculty member describes the importance of working across departments for learn- 
ing: “Physics has always been way ahead of other departments at our institution. Now, what 
they're doing is spreading to other departments because, through this initiative, we had to 
begin to work across departments, and share information and learn from each other.” 


Working as teams: Campus project teams typically included faculty across multiple depart- 
ments along with administrators and staff from academic affairs. By connecting individuals 
and groups that typically do not interact — central administrators with individual teachers for 
example — project teams sometimes led to insights about implementing, supporting, and scal- 
ing effective teaching on campus. One 

administrator described the learning that Most of the respondents 


occurred from these interactions: What it acknowledged that working 
meant was anybody who was involved in 


the AAU [Initiative] really can't be isolated. across units or departments 
They're not allowed to be isolated. They to foster improved undergraduate 


have to talk to the teaching center. They d ti tivoical bet 
have to talk to the teaching center for the education was not typical betore 


evaluation. In these discussions, particularly the AAU Initiative. 

about their evaluations, they rethink their 

teaching.” This same person goes on to say: “I really think it’s the integration and working 
together. If you integrate these things together, the person cannot be solitary. They can’t be 
isolated. Once they’re not isolated, they start talking to other people. It’s just, ‘Oh, | had that 
same issue.’ Or, ‘Oh, is this how you solved that? Look what happened in my class. Did that 
same thing happen for you and your class?’” Most of the respondents acknowledged that 
working across units or departments to foster improved undergraduate education was not 
typical before the AAU Initiative. 


Site visits: Site visits were important sources of both influence and learning. In addition to 
gathering data, AAU staff brainstormed with project teams about solutions to local chal- 
lenges. AAU staff also shared information about instructional reforms on other AAU cam- 
puses. The activities were instrumental to learning at the project sites. An administrator com- 
mented on the value of the site visits: “One thing | want to highlight is, the site visits, | think 
that was an opportunity for us to get out of our bubble, just doing things and implement- 
ing. | feel like we really engaged ideas differently when the AAU came here.” 


13 Respondents portrayal of change as a combination of learning with strategies such as networks, systems, and influence was quite 
common as discussed in the next section of the report. 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 61 


Visiting another site: Very few campuses took advantage of the opportunity — provided by 
AAU funds as part of the Initiative — to visit another project site. Those that did found the 
experience helpful in adopting a new curriculum, changing pedagogical practices, and using 
data analytics. One faculty member talked about visiting another site: “We got a whole new 
perspective when we went to MSU. I’m really interested in their center for scientific teach- 
ing. It really helped expand our thinking around assessment. I’m coming back to our campus 
and saying, ‘We got these ideas at another AAU campus.’ That opened up faculty to consid- 
ering the ideas we brought back.” 


AAU STEM Network meetings: Points of contact (POC) and project site team members 
learned from one another at AAU-related network meetings. Sometimes the focus was on 
local reform, other times on becoming better prepared for future competitions for funding. 
One POC described the learning resulting from network meetings: “Having heard from the 
project sites at the meeting, we've adopted improving teaching as a part of our strategic plan 
and it’s a major priority. We are starting an initiative to bring in discipline-based education 
researchers. And we have adopted some of the teaching practices such as assessment and 
clickers.” Another POC stated: “We're doing a lot with student data now that we borrowed 
from UC Davis. We're really focused on analyzing student pathways. We also adopted some 
evidence-based teaching practices, particularly, active learning.” And yet another POC: “We're 
working on a lot of different fronts right now. We started the process to hire more disci- 
pline-based education researchers. We began expanding our data analytics, we were already 
doing that but using more of the ideas from UC Davis. We're also enhancing more of our 
classroom spaces — using more technology — using some of the ideas from UPenn. And we 
are doing a lot of new curricular work.” These findings suggest that adoption of new prac 
tices may be driven by influence and competition as much as by learning. 


Peer learning: |nterviewees universally appreciated the opportunity to learn from peer insti- 
tutions. The faculty and staff at peer AAU institutions understood key components of instruc 
tional reform: the tenure and promotion system, departmental cultures, value systems around 
teaching, and infrastructure. Some believe that the continued conversations among faculty 
and staff at research universities are creating change and perhaps learning: “I think by repeat- 
edly bringing together some R's that are really thinking about this, thinking about data, think- 
ing about instructional change, what has happened are collaborations and exchanges.” 


Norms that reinforce examination of teaching practices: |ndividuals reported that 
changing cultural norms — such as having high expectations for quality teaching — was key 
to scaling and sustaining of excellence in teaching. For example, one interviewee suggested 
that, “We need ongoing cultures that support teaching excellence.” Another stated that, 
“Everyone should feel like they need to innovate their teaching.” Interviewees found AAU's 
Initiative helpful in altering logics and norms, which were important to support individual 
and campus learning. Changing the conversation to frame change as “ongoing” and “con- 
tinuous” would increase the effectiveness of the Initiative. 


Feedback from annual reports: AAU required annual reports from project sites and pro- 
vided feedback to campuses based on those reports. Some interviewees found this feed- 
back helpful. For example, one project team member responded: “They were really helpful 
at meetings, site visits, and even with the annual reports. We got questions about ways we 
could engage leadership more and how we were going to sustain the efforts after AAU 
funding went away and information about another campus that was engaged with a simi- 
lar pedagogical effort.” 
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Framework: Vision and Learning Guide. AAU developed the Framework document to pro- 
vide a vision for the AAU campuses — a sense of what excellence in teaching at AAU institu- 
tions might look like. It also presented STEM improvement as an institutionally situated chal- 
lenge that required support and attention at all levels of the institution. AAU developed the 
concept of Framework in part by reviewing the APLU Rubric for teacher education, although 
AAU's Framework was more conceptual and less detailed. 


AAU sought substantial feedback on the initial Framework (as well as with many other aspects 
of the Initiative). The initial draft Framework was refined based on feedback (a strong practice 
for obtaining input) provided principally by POCs at 42 AAU member campuses. The final ver- 
sion of the Framework reflected a change from a detailed and proscriptive set of instructions to 
a more conceptual document about implementation and scaling change. 


AAU expressed the Framework in both diagram and text formats and made it available as an 
online resource. AAU aligned its RFP for project sites with the Framework. AAU also used the 
Framework to organize meetings, reports, and most project deliverables. It became the foun- 
dation for all communications materials related to the AAU STEM Initiative (included in the AAU 
website). The Framework was, in many ways, omnipresent. Yet many interviewees did not 
describe the Framework as a helpful tool for implementing and scaling STEM instructional 
reforms. Even some members of the AAU STEM Advisory Committee were unable to enumer- 
ate the elements of the Framework. 


The Framework was rarely brought up in discussions about effective practices or strategies by 
project sites. When probed, respondents indicated only a vague understanding of the 
Framework: “| know probably about the framework. It has circles. And | know probably the big 
goals. | don’t really know all the details as well as | should, but | know the framework is about 
different ways that you can affect change in teaching.” 


One group that consistently found the Framework useful: campus champions (including the Pls 
of project sites). At each project site — and in the Network as a whole — one or two individuals 
were passionate about and champions of innovation in STEM teaching. Many of these individ- 
uals were familiar with the literature on STEM reform strategies. They were the most likely to 
note value in the Framework. One participant explained: 


“| actually think that having worked with and thought through the Framework, it was really 
helpful. | think it was really important because lots of times when faculty think about 
making changes, they think about curriculum development. Then | as a faculty developer 
will go and say, ‘Okay, I'll work with you, faculty member, to make your curricular develop- 
ment.’ I'll be honest, at the time they gave it to us | was like ‘Oh man!’ But now | see how 
helpful it is. We're doing curriculum development and people go off into their own little 
courses, right? Because we have the faculty development in the cultural change it’s really 
making a big change across university.” 


The results suggest that the Framework was most influential for project team leaders, less so 
for other participants. 


The Framework seemed best understood by project leaders and campus champions; it seemed 
less successful as a general tool for learning about change for other participants on project sites. 
Several possible explanations for this disparity exist. First, each campus was asked to submit a 
project proposal focused on one aspect of the Framework while at least acknowledging all 
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levels of the Framework. Campus participants may only have become familiar with the compo- 
nent of the Framework included in their project site proposal; campus participants seemed quite 
knowledgeable about their own projects. AAU did not ask proposals to address the entire 
Framework because it might deter campus teams from applying. Using the Framework as a 
broad educational tool might require additional communication mechanisms. 


Second, the Framework incorporated elements of both improving teaching and cultural change 
(at the department, college, and institution). The blending of a vision for improving teaching 
with a process for creating and scaling change might have confused some participants. As a 
potential learning tool AAU might have been better served by separating discussions of peda- 
gogical reform from the process of change. As one respondent stated: “| find the framework 
overly complicated. It’s working at all these different levels, it talks about culture change, and 
then it talks about specific practices. | just tend to get lost in it.” 


Third, when analyzing data based on role differences — faculty member vs. administrator for exam- 
ple — there was a clear trend for faculty members to have difficulty understanding the role of the 
Framework’s third circle — cultural change — in reforming pedagogical practices. Even a few adminis- 
trators had difficulty linking the components of the Framework into a coherent vision for local reform. 


The challenge for AAU is to distinguish the Framework as a tool to promote change by influ- 
encing campus champions and project leaders (e.g., project leaders used the Framework in shap- 
ing project activities) from the Framework as a tool to educate a variety of participants on mem- 
ber campuses about the reform process (which was much less successful). The latter function 
likely requires a different dissemination strategy, one based on careful delineation of the roles 
of AAU (e.g., influence vs. learning) in campus reform. 


Why Learning was Challenging. The Initiative was less successful in its role in fostering indi- 
vidual and institutional learning than in, for example, influence for several reasons. Some inter- 
viewees believed that AAU through its Initiative first had to help establish new norms about 
undergraduate teaching, after which improved instructional practices would emerge. In addi- 
tion, some factors emerged from study data as potential barriers to individual and institutional 
learning: perceiving practices from other institutions (or even other departments at the same 
institution) not being applicable, competition between campuses, the need for local data for 
claims of effectiveness to be accepted, AAU's limited history in directly engaging in institutional 
change (especially for teaching), and lack of meaningful venues for deep discussion. 


“Not invented here” - unique institutional culture: The AAU STEM Initiative faced a par- 
adox. AAU acknowledged the importance of institutional context in reforming undergraduate 
STEM education while at the same time encouraging institutions to learn from each other and 
from research on teaching and learning. Respondents sometimes found that the local culture 
inhibited reforming undergraduate teaching, and that challenging local norms rather than hon- 
oring them was crucial to change. AAU support of local context as a driver of change often left 
change agents in a bind trying to navigate local pressures. The phenomenon showed most 
clearly when opponents of reform argued that changes created in other units on their campus 
or at other universities would be inappropriate for the local environment. 


Most interviewees described AAU’s “sensitivity to research university culture” and unique insti- 
tutional mission as positive aspects of the Initiative. One administrator noted: “Yeah, | think they 
recognized early on — They know all these institutions are different. They seem to honor in a 
way that | really like, the fact that the people that are at those institutions and the ones that are 
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in the department are the ones who can see the greatest opportunities and make them fit their 
context. | really like that.” As one faculty member described: “No, we really haven't adopted 
any specific ideas or practices from other campuses. Our campus is quite different from the oth- 
ers. But | enjoy hearing what’s happening on other campuses.” Or “I know that these are AAU 
institutions, but we're quite unique. We're smaller, our curriculum is more fluid and interdisci- 
plinary, our faculty are different, so | just don’t see a lot of similarity.” 


In contrast, some interviewees found an overemphasis on institutional context and culture could 
be used to deflect change: 


“There is a lot going on in the liberal arts colleges in terms of defining excellence in 
teaching in STEM. The fact that our institutions won't even look at this work because 
they don’t think it’s important for research universities or that research universities 
are so different is not helpful. This good work in other institutions [liberal arts col- 
leges] are not highlighted within the Initiative, and in fact the idea that research uni- 
versities are unique is constantly repeated.” 


Another administrator suggested how the focus on institutional context might prevent adop- 
tion of good ideas: 


“Maybe, in a tiny bit of irony, the HHMI thing that we’re involved in is really trying to 
understand whether the specific initiative that was introduced at University of 
Maryland Baltimore County, UMBC, can be successful at other campuses. Whether 
that can be taken from a blue collar regional school, where they've been just extraor- 
dinarily, extravagantly successful with preparing students for careers in science, espe- 
cially minority students, to other places. But there is this feeling we have to prove it. 
So, we are trying to translate that [improved teaching practices] into AAU type 
schools. There’s some irony about that as well. That on the one hand, we're happy 
for our association with AAU and learning from other schools. Then, in a related proj- 
ect, we're talking about a school that wouldn't, in a million years, be in the AAU.” 


These findings suggest that paying attention to institutional type and unique context must be 
balanced by emphasizing the value of and transferability of ideas from other contexts. 


Competition: We noted previously that competition served in some cases to stimulate local 
teaching reforms — for example, implementing reforms because peer competitors were doing 
it. However, adopting approaches from other institutions became problematic because bor- 
rowing from other sites implied an equal or : 

higher value to external norms than to local Adopting approaches from other 
ones. One faculty member described how institutions became problematic 


competition could adversely affect the because borrowing from other sites 
sharing of ideas to improve undergraduate 


teaching: “I think we all just came in implied an equal or higher value to 
strongly believing in our ideas and maybe external norms than to local ones. 
we felt borrowing ideas from the other 

campuses would somehow support them and we are competitive with each other.” 


Competition also affected sharing data across institutions. Early on campus teams were reluc 
tant to share data about their teaching practices with peer institutions. AAU project staff 
were concerned early on about campus competition preventing participants at annual 
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gatherings from problem solving together, learning and sharing, and might lead them to miss 
out on opportunities within the broader network." It is possible that a fear of losing a com- 
petitive edge adversely affected sharing information across project sites. This concern seemed 
especially true of student retention rates and learning outcomes. At the curricular level, how- 
ever, competition did not prevent some sites from seeking solutions implemented elsewhere. 
As one example, the University of California at Davis considered adopting the Chemical 
Thinking curriculum developed at the University of Arizona. 


Distrusting research and evidence from other contexts: A particularly onerous form 
of “not invented here syndrome” centered on distrust of research results about pedagogical 
reforms. This distrust was as evident between departments in the same institution as between 
institutions. Some project participants wanted to collect their own data, even if that was time 
consuming and resource intensive, because they believed their instructional environment was 
unique. As one example: “We would have to conduct a study before considering ideas from 
other campuses. So that has made us leery of considering the ideas — the time it takes to con- 
duct a study.” 


AAU not known for leadership in teaching and learning: AAU (and many of its mem- 
ber institutions) have a limited track record in supporting improvements in undergraduate 
education. Although mentioned by only a few interviewees, it may be that its “newcomer” 
status to undergraduate reforms affected AAU'’s ability to shape learning. One respondent 
exemplified this belief: “There are so many great examples of phenomenal teaching in STEM 
at liberal arts colleges and even community colleges. This just hasn't been a strength area for 
AAU, so you wouldn't imagine that this is an area — | guess | can see they would have some 
struggles here.” We note that this perception is held by only a few interviewees and is not 
necessarily reflected in the literature on DBER, which contains many examples from research 
universities. Perhaps this perception is in part a function of AAU’s recent entry into the under- 
graduate education policy arena. 


Conclusion. AAU's role in learning was seen by respondents as less important than other roles 
the organization played. Evidence showed some examples of adoption through learning includ- 
ing data analytics, curriculum and learning spaces. We note that the Initiative, in part by design, 
has paid more attention to the roles of influence and prestige on promoting educational reform. 
It has had less success in achieving reform through individual and institutional learning. 


14 Over time this concern by AAU project staff dissipated. 
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Value of Multiple Theories/ 
Strategies of Change 


AAU's approach in the Initiative drew from multiple theories to inform a complex change strat- 
egy. This approach is consistent with a growing body of literature showing that systemic change 
efforts are enhanced when actors use several change approaches consistent with their assets 
(Boyce, 2003; Kezar, 2013; Sporn, 1999). In this section, we summarize the interplay of these 
theories and examine how certain strategies can embed multiple theories of change/action. 


Harnessing Multiple Theories. As noted in the literature review, AAU's initial conceptualiza- 
tion of the Initiative reflected several different theories of change. Most prominent were theo- 
ries of institutionalization and, to a lesser degree, broader systems theory (examining the inter- 
play of various parts of the higher education system to create and influence change), networking, 
and organizational learning. Although not described in Initiative documents, the Initiative also 
drew from institutional theory, particularly in viewing AAU through the lens of influence and 
prestige. Employing multiple theories of change is important because theories might vary in 
their relevance depending on the type of change, context for change, challenges related to the 
change, or the actors involved. 


Institutional theory: AAU's prestige, success in using influence strategies, and reframing 
institutional logics all suggest the importance of institutional theories (IT) of change. IT 
emphasizes change as a result of external forces and the reshaping of cultural norms and log- 
ics. Interviewees confirmed that AAU’s most important role in scaling change came from 
helping to reshape cultural norms through its prestige and influence. 


Network theory: Because networks can influence scale and foster change, networking is a 
key concept in institutional theory. Yet networking is its own theory of action as well. 
Respondents found AAU’s role as a network creator and supporter to be an extremely suc 
cessful strategy that built on AAU’s strengths and assets. Evidence throughout this report 
supports AAU’s role as a network developer, and the functions these networks served in 
helping to scale changes. 


Organizational learning: Theories of organizational learning suggest that learning is the 
key driver for change. From this perspective, challenging existing assumptions, reviewing new 
information, and undergoing an inquiry process is required to promote new norms and ways 
of working. AAU facilitated learning, in some instances, particularly through networks, which 
demonstrates a kind of synergy between strategies. Despite some evidence of success, AAU 
was less successful in employing learning strategies than in networking and other pres- 
tige-based strategies. 


Systems theory and institutionalization: |nterviewees noted the importance of scaling 
reforms at multiple levels, which is consistent with systems theory. In particular, interviewees 
discussed the efficacy of strategies within different parts of the system, as well as an ener- 
gizing process across strategies at different organizational levels. 


Institutionalization theory emphasizes the importance of leadership and infrastructure in 
implementing and scaling change. Institutionalization can also be seen as a facet of systems 
theory in that the institution is a system of various parts which must be coordinated to achieve 
long-lasting change. Systems theory focuses on the interrelationship of various institutional 
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behaviors and policies — tenure and promotion requirements, professional development, insti- 
tutional commitment, and classroom assessment — in fostering reform. 


Our study identified three important findings related to multiple theories of change. First, we 
identified the value of using several theories of action to create synergy to amplify the change. 
Second, we illustrate how national organizations can make use of their organizational iden- 
tity, assets and niche in determining which theories of action to use. Third, intentional deploy- 
ment of theories can assist in achieving policy reforms. As a prestige organization, AAU was 
able to leverage its influence (a tactic deriving from institutional theory) toward change and 
create new institutional logics that helped reshape values. AAU also employed an open sys- 
tems approach by leveraging its prestige and influence to gain support from such prestigious 
organizations as the National Science Foundation and the National Academies of Sciences to 
create support for improving teaching. AAU is one of the few groups with the prestige to 
align many different players/organizations within the higher education system and to create 
a powerful and aligned network of action. In general, AAU has the prestige to convene and 
organize networks. People want to attend its meetings. Using these various approaches 
together helped AAU’s success in scaling change to date; relying on a single theory of change 
would have been too limiting. 


Evidence suggests that AAU’s reliance on multiple theories was important to the success of 
the Initiative. No single theoretical model was sufficient to guide such a complex reform 
effort. Overall, institutional theory and its focus on influence seemed a better fit for the AAU 
Initiative than theories of action related to 


institutionalization of changes, especially at No single theoretical model 


the institutional level. Theories of organiza- + ¢ ‘ 
tional and individual learning seemed least was sufficient to guide such a 


well suited for the AAU Initiative. complex reform effort. Overall, 
a a institutional theory and its focus 

Had AAU made its reliance on institutional infl fi 

theory more explicit it might have made for on influence seemed a better fit 


more effective implementation at local insti for the AAU Initiative. 

tutions. Given the lack of explicit external 

articulation, planning, and strategizing related to influence strategies or institutional logics, 
AAU’'s use of influence was not as robust as it could have been. This study points to the impor- 
tance of making theories of action explicit so they can be used for planning purposes. 


Strategies that Reflect Multiple Theories. In this section, we review four strategies that 
reflect multiple theories of change. It may be important for organizations trying to scale change 
to align implementation strategies with the various change models implicit in their overall 
approach. In the Initiative these strategies could include influence, learning, challenging assump- 
tions, networking, resetting norms, exploring values, developing leadership, or mapping prior- 
ities onto practices and policies. 


Site visits: The AAU site visits exemplified an activity reflecting multiple theories of change. 
Interviewees said site visits helped facilitate change in various ways. 


Influence. Respondents described site visits as instrumental in influencing change because 
the AAU staff met with presidents and provosts and other key leaders, which brought 
attention to the Initiative. An administrator described the value of the site visits: “Each 
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time we have a site visit we've really been invigorated. After our last site visit for example, 
| was able to get administrators’ attention on using data analytics - something | was unable 
to do in the past - and to think again about setting up some kind working group around 
promotion and tenure.” Another respondent said: 


“Emily as well as the folks at AAU have been part of pushing the conversation 
about how to create a climate supportive of teaching with the leadership at 
my school. It’s not as if we're ignored by the leadership, but it really helps 
when they come and create more awareness for the programs we are trying 
to create. They talk to our deans and provosts. And it’s not been without 
notice. | hear things like ‘Wow, it’s really interesting what you guys are 
doing.’ | just had this conversation with those folks from AAU. So, they're 
very impressed. So, when they come and visit and talk to our chairs or talk to 
the deans or talk to the provost, that really matters. That’s the thing to keep 
things moving.” 


Project teams were able to use the AAU brand name to get resistant faculty and adminis- 
trators to attend meetings. The site visits also seemed to help project teams make head- 
way in their work. 


Sharing practices and learning. AAU also used site visits to create learning among project 
teams, sharing best practices from other campuses to get them to explore or consider new 
approaches and brainstorm implementation strategies. Additionally, AAU staff encouraged 
sharing ideas across departments by describing a change in one department (e.g., biology) 
to influence faculty and staff in another department (e.g., physics). Site visits also provided 
the opportunity for learning by allowing teams to brainstorm difficult challenges: “We 
wanted to use the model of apprenticeship from North Carolina, but we're really worried 
about the tenured faculties’ reaction. But in talking with AAU, we were able to think that 
through.” Further, AAU staff challenged values and priorities — asking, for example, have 
you reconsidered your hiring criteria? The researchers observed AAU staff repeatedly ask- 
ing questions that were aimed at prompting learning. 


Networking. AAU staff used site visits to expand project site networks, connecting teams 
to faculty and administrators on other campuses. A faculty member noted: “Each time 
they visit, we have been able to identify [through AAU staff] another campus or person 
working on a similar challenge that we could contact.” These respondents also found that 
the presence of AAU representatives on their campus helped persuade otherwise reluc 
tant faculty and staff to attend events. This example shows the synergy of connected strat- 
egies where AAU’s “influence” strategy leads to a broader campus network. 


System theory. Lastly, AAU generated systems thinking by asking individuals on a campus 
to reflect on their progress in relation to the Framework. Throughout a site visit, AAU staff 
prompted questions from administrators that focused on institutional commitment and 
infrastructure. Similarly, they asked faculty and department chairs about the role of infra- 
structure in helping or impeding their work. AAU staff sometimes communicated perspec 
tives about work across the system. For example, telling administrators how their improve- 
ments in infrastructure helped support the campus project team in improving 
undergraduate STEM instruction. 
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RFP and design of the project sites: The RFP is another example of a strategy reflecting 
multiple theories of action. The RFP was noted by various interviewees as well-structured and 
strategic. 


Influence. First, the RFP reflected an influence strategy that succeeded in getting key 
administrators across many AAU campuses to think about improving undergraduate teach- 
ing in STEM courses. Many of the campuses that did not receive funding for a project site 
talked about the usefulness of the proposal process in initiating campus action: “So, we 
didn’t even get the funding, but we still have several efforts that | think are to create an 
institution-wide change. Normally, we would've had separate initiatives, but the RFP 
pushed us to start thinking about our work in teaching reform differently and to consider 
more institutional infrastructure that’s needed. Without the AAU Initiative, that just 
wouldn't happen.” In sum, the RFP and AAU’s general influence alone helped spur several 
AAU campuses to pursue reforms in undergraduate STEM education. 


Organizational learning. Many interviewees said the AAU RFP made their campuses (fac 
ulty and administrators) think about STEM reform in new ways, noting how it promoted 
learning and systems thinking. For example, faculty and administrators started thinking 
about overall institutional support for educational reforms instead of relying mostly on 
individual faculty members in departments. One faculty member described the learning 
that resulted from the RFP and proposal process: “| think it really goes back to the Initial 
request — its design. We had to shift our thinking from individual pockets of work spread 
around to an institutional effort.” 


Systems theory and institutionalization/networking. The RFP asked campus leaders to com- 
mit resources to the proposed local project and asked for a letter of support from the pres- 
ident or chancellor. The RFP also challenged institutions to create teams of individuals 
across faculty, staff, and administrators to implement the proposed project. In this way 
the RFP required both top-down and bottom-up commitment and support. By fostering 
leadership across the institution, AAU was enabling (or in some cases reinforcing) systems 
change to take hold. By encouraging senior leadership involvement along with key faculty 
and staff, AAU set the stage for an effective internal network to work toward change. 


Annual meetings: The annual meetings and workshops were highly regarded by respon- 
dents. Participants enjoyed the opportunity to get together in person, network, share and 
learn information, and consider the various strategies needed to support sustained change. 


Networks. As noted in Chapter 8, annual meetings were essential to develop a larger net- 
work for undergraduate STEM reform. Annual meetings helped develop and later enhance 
connections across campuses between faculty in particular disciplines, many types of 
administrators, and champions who were passionate about STEM reform. 


Organizational learning. Many interviewees also said the annual meetings were important 
opportunities for learning. As one faculty member described: “! think the most successful 
aspect were the meetings. They provided us a chance to reflect on our work, to learn about 
what other people were doing, to hear about what’s happening at the national level, and 
recommit to the work.” Some interviewees described specific outcomes from the meet- 
ings, including adoption of practices from other campuses such as data analytics devel- 
oped at UC Davis. 
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Influence. Holding annual meetings at the AAU national offices or at another location in 
Washington, D.C., as well as bringing in prominent speakers to talk with attendees also 
influenced participants. According to one respondent: “It’s hard to exactly pinpoint, but 
there was something about being in Washington, D.C., and mixing with people | would 
never get a chance to interact with and feeling like this is so different from other meet- 
ings | might go to.” 


Systems theory and institutionalization. Because AAU used the Framework to organize 
meeting activities, the annual meetings also reinforced key elements of a systems theory 
of change and institutionalization. For example, meeting sessions focused on promotion 
and tenure, improving professional development, and assessing learning outcomes. AAU 
also brought in influential speakers from prestigious organizations and national leaders in 
STEM reform. 


Media attention to the Initiative: The media and press coverage of the AAU Initiative, par- 
ticularly of specific campus activities, reflected multiple theories of action. 


Influence. Bringing media attention to campuses was an influence strategy. It helped repri- 
oritize the Initiative on some campuses. In a few instances media attention brought in addi- 
tional resources and infrastructure. One faculty member stated: “I think having the admin- 
istrator see the story in a major press was part of the reason we got additional resources 
for the effort, and so something that seems basic can have a much bigger impact.” Media 
also stimulated reform by leveraging the inherent competition between institutions. When 
some campus leaders saw their peers gaining media attention for STEM reforms, they 
wanted to respond by developing or publicizing their own reforms. 


Learning. Campus leaders often engaged in learning when they developed stories for the 
media. Journalists asked pointed questions that helped campus project participants and 
senior leaders think about their reforms in more detail. Many interviewees suggested that 
working on a press release or working with AAU public relations staff allowed them to 
reflect on their overall work and bring insights back to the project team. One administra- 
tor stated: “This might seem like a small issue, but in having to talk to people off campus, 
in the media who are not in STEM, it makes me push my thinking about what we are doing. 
| need to be more explicit in how | describe it. | find myself having insights into our work.” 


Networking. A few interviewees talked about reaching out to personnel at another cam- 
pus after seeing a news article about STEM reforms. In such cases, faculty members 
reached out to peers at the other campus. Administrators also talked with their peers at 
other institutions. 


Conclusion. AAU’s STEM Initiative was a complex change process. It leveraged several approaches 
to scale change across different stakeholders, campuses, networks and organizations. Strategies 
designed to share best practices were most effective with faculty and campus points of con- 
tact. Influence was more effective with provosts and deans. Networking was especially helpful 
to local champions for STEM innovation in teaching. Reform strategies can increase impact by 
embedding as many approaches to change as possible, as demonstrated by the various theo- 
ries of action embedded in the RFP/ project design, annual meetings, site visits, and media/pub- 
lic relations. 
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Differing Group Perceptions 
of AAU’s Role in Improving 
STEM Learning Environments 


Although there was agreement among interviewees about the most effective roles of AAU in 
scaling change (influence, networking, and resetting institutional logics/norms), group differ- 
ences were also apparent among collaborators, administrators at the project sites, faculty at the 
project sites, and points of contact (faculty or administrator). These differences raise the ques- 
tion of how to serve different groups simultaneously or even whether it is possible to serve mul- 
tiple groups if their needs differ substantially. One option is to focus primarily on the constitu- 
ent groups for which an organization has maximum impact. The findings also suggest the 
importance of AAU framing issues differently for constituent groups with quite different needs 
and expectations. Being aware of differences when working with multiple groups is critical to 
scaling change. Furthermore, understanding different group perspectives is important to iden- 
tify where an organization such as AAU can most meaningfully engage with its constituents. 


In the sections below, we review some of the key differences by groups interviewed.'> 


AAU as Prestige Organization. Senior administrators were much more aware of AAU as a 
prestige organization than were faculty members. These administrators were also knowledge- 
able about the role of AAU in shaping the overall landscape of higher education and its history 
of shaping the overall sector. As an example, one administrator stated: “Well, the AAU gets a 
lot of respect as an organization. It’s the top research universities and the fact that the AAU is 
actually doing a project in STEM signals to a lot of people something is different. | think it adds 
a lot of weight to the initiative and to the attention that’s being paid to it among the top 
research universities in the country.” Some faculty members admitted that they had never heard 
of the AAU before that the Initiative but have since learned it is important to their institution. 
These faculty members thought their colleagues remain largely unaware of the AAU and its 
influence. One faculty member stated: “| know this [AAU] is a big thing among administrators, 
well | have become aware of that. But for faculty, they might have heard something about being 
an AAU institution, you know that is prestigious, but they do not know an AAU organization 
even exists.” Department chairs, initially generally unfamiliar with the AAU, sometimes found 
out about AAU through discussions with senior administrators. Said one chair: “I’ve had the 
opportunity to learn more about the AAU, but most of the faculty | interact with — they just 
don’t know what it is.” 


The Initiative often worked with teams comprised largely of faculty and department chairs, 
groups often not sufficiently informed about AAU to leverage the AAU brand to improve STEM 
instruction. A few project sites and points of contact did develop strategies to take advantage 
of AAU’s status in promoting local STEM reforms. From this perspective AAU might increase its 
impact on STEM reforms by engaging in a communication plan to raise awareness of AAU as 
an organization among individual faculty and administrators. Such an initiative, though, seems 
a questionable investment by AAU of time and resources. 


Understanding of Framework. As discussed in Chapter 9, collaborators and senior adminis- 
trators were more familiar with AAU’s Framework than the faculty. The faculty and some 


15 There may be additional differences, as we interviewed a selective set of groups. 
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administrators, including department chairs, often struggled to translate the element of cultural 
change in the Framework into strategies to better align infrastructure with undergraduate 
instruction. These differences in perspective suggest that AAU (and other organizations that 
engage faculty and mid-level staff and administrators) may need to find ways to help STEM 
reformers position their work in the broader institutional environment. 


Differences in Defining Impact or Scale of Initiative. Participants varied substantially in how 
they defined key outcomes such as impact and scale. Collaborators were much more likely to refer 
to impact as actions affecting all AAU institutions as well as the national dialogue about under- 
graduate STEM reform. Senior administrators described scale in terms of institutional change; they 
were unlikely to consider other AAU institutions in their definition. Department chairs often lim- 
ited their definition of scale to a few faculty members in a department or sometimes across depart- 
ments. Faculty members typically viewed scale as reform across a department especially by engag- 
ing more members of the department in the 


reform effort. Even members of the AAU lead- When AAU spoke about scaling 


ership team differed in their understanding of h h tit t h d 
impact or scale. Some believed that scale was change eacn consttuent group hear 


about individual institutions changing their cul- that statement in different ways. 
ture. For others scale referred to all AAU insti- 

tutions altering their culture. Others considered scale as both impacting the national discourse on 
STEM reform and reforming AAU institutions. In this context, when AAU spoke about scaling 
change each constituent group heard that statement in different ways. These differences in per- 
spective sometimes adversely affected the use of a systems approach for reform (see Chapters 4 
and 5). In particular these different perspectives about scale reduced the potential synergy of local 
actions because the participants were not always moving toward the same objectives (noted in 
Chapter 4). For example, departments on the same campus sometimes differed in their views 
about the need for STEM reform and about the strategies required to achieve it. 


Beliefs About Who Should Lead Change Efforts. The faculty differed with senior admin- 
istrators when describing leadership in undergraduate STEM reform efforts. Faculty members 
tended to emphasize a bottom-up view with leadership roles for influential faculty, department 
chairs, and the department as an entity. From this perspective faculty and departmental lead- 
ership is required for eventual institutionalization of reforms. Senior administrators emphasized 
the need for shared leadership between the faculty and middle-level administrators such as 
chairs and deans. Many senior administrators did not assert the importance of their own role in 
leading change. Many faculty respondents cited a lack of senior administrator involvement in 
the change process as a potential barrier. Yet the faculty did not envision a primary role for these 
administrators in their own “change model.” The apparent lack of leadership at multiple levels 
could be a barrier to institutionalizing reforms. It is also possible that the form of support from 
senior leadership, such as providing funds and renovating classrooms, was less apparent to proj- 
ect participants than the involvement by faculty and staff implementing the reforms. 


Role of AAU. Respondents differed in their views of the role of AAU in the Initiative based on 
the role, their work on the project, and their understanding of scale. Interviewees tended to 
view AAU through the lens of their own needs and challenges. Faculty members described ways 
that AAU could help them develop tools for course and curriculum reform and find ways to help 
modify reward structures. One faculty member commented: “I'd really like the AAU to create 
an extensive curated website with examples of new pedagogies and curriculum, by discipline, 
nationally. Nothing like this really exists. NSF tried to do this, but it is not up to date and includes 
a lot of non-AAU institutions.” Another noted: “It might really helpful if they sponsored some 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 73 


professional development or workshops across AAU institutions that focused on teaching and 
improving teaching.” Faculty tended to view AAU as a potential provider of services needed to 
meet on-the-ground challenges. Administrators of all types were more likely to emphasize AAU's 
role in helping garner additional resources for projects similar to the Initiative; including funds 
to improve teaching facilities, support for data analytics, assessment tools to measure teaching 
effectiveness, and providing examples of other successful reforms. As one administrator stated: 
“| have appreciated their work on uses of data. We have had a few presentations. And | think 
they are working on measures of teaching. These efforts are both really useful.” 


A realistic view of AAU demonstrates that it cannot respond to all of these individualized group 
and campus needs; it lacks the organizational capacity to do so. Even if it had the capacity, it is 
unclear that focusing heavily on local needs is an effective role for AAU. One way to determine 
the best match between campus needs and AAU capacity and strengths is to conduct a needs 
assessment to identify unmet needs, and then identify where AAU might be the appropriate 
organization to help and where it is not the appropriate organization. AAU could also use the 
needs assessment to identify possible partnerships or broker with other organizations that might 
be better suited to helping campuses meet local needs. For example, connecting faculty at proj- 
ect sites to professional development organizations such as POD might help faculty obtain sup- 
port for pedagogical reforms. As another example, AAU has strongly supported data analytics 
work, but other organizations are better suited to helping member institutions use data analyt- 
ics on their own campuses. 


Conclusion. This chapter discusses the importance of comparing AAU strengths and capaci- 
ties with local campus needs. We recommend using a needs assessment to help AAU and its 
member campuses determine the most strategically effective roles for AAU and its member 
institutions. In addition, making clear the boundaries of AAU’s reach — whether by describing 
the limits of its resources or clearly stating its range of responsibilities — will help clarify AAU’s 
role as the Initiative goes forward. Also important is developing strategies that take into account 
differences in role group perceptions about the change process. 
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Scaling Reform 


One of the goals of this report was to understand AAU’s ability to implement strategies shown 
to be effective in scaling reforms, including creating forms for deliberation, developing external 
supports (often provided by an intermediary organization), and creating and supporting net- 
works. This chapter summarizes the evidence about the use of well-established factors in scale 
in AAU's Initiative. 


Previously, we outlined three key aspects of theories of scale: deliberation, networks, and exter- 
nal supports. Opportunities for deliberation and dialogue among professionals — such as conven- 
ings, conferences, and meetings — develop motivation, interest, and ownership (Elmore, 1996; 
Kezar, 2011). Through dialogue, norms and values are explored and changed as people accept 
new ways of behaving. Networks connect people to others with similar ideas and also provide 
change agents with information, incentives, and social capital. These assets facilitate the change 
process by providing knowledge about how to overcome barriers or foster working relationships. 
Networks also reduce risk-taking by having groups of people experiment together. External mech- 
anisms of support can take many forms ranging from concept papers and frameworks to fund- 
ing, awards, and recognition, which often are necessary to help sustain change agents in the 
face of entropy or negative social dynamics. These strategies acknowledge that individuals (or 
even individual campuses) cannot successfully act alone to scale changes, and that their environ- 
ments must be reshaped to support ongoing change. An individual campus is unlikely to act out- 
side the norms, messages and incentives developed within the broader enterprise of higher edu- 
cation, which may be especially true for the major universities in AAU. Disciplinary societies, 
national organizations, and regional groups all shape and influence campus behaviors, including 
individual faculty behaviors. Aligning messages among groups provides motivation for change. 


Deliberation. The AAU’s work to create the deliberation necessary for change was incomplete. 
AAU worked to recreate institutional logics and language that are often part of deliberation pro- 
cesses (see Chapter 7). However, AAU needed more input from various stakeholders to create more 
consensus about the focus of the Initiative. Most interviewees did not feel that this was accomplished 
through the Initiative. We found little evidence from the observations of deep deliberation. 


In our observations and interviews on campuses and at network meetings we found the focus 
of project sites mostly on implementation of project activities, and to a lesser degree on insti- 
tutionalization. Much less attention was given to deliberation about the meaning of the reform 
process and especially changing local cultures. As one project site interviewee noted: “Most of 
the project sites just focused on implementing the work, and very little discussion is happening 
across campuses about improving teaching.” Most of the project sites involved small groups of 
faculty and a few administrators, which often seemed unable to generate broader discussion of 
teaching across campus. 


A few interviewees from AAU STEM Network campuses mentioned some new conversations 
that occurred as a result of being part of the AAU Initiative: “Before the initiative we had very 
little work on evidence-based teaching and now we're having discussions about active learning 
and need for space to experiment with to do this work.” 


Some interviewees also commented that the primary focus in annual meetings was on informa- 
tion sharing and implementation strategies, less on deliberation about the reform process: “I 
like hearing what's happening on other campuses, and at the national level, it’s interesting but 
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not leading to deep conversations.” Another point of contact commented: “The meetings lack 
the kind of depth that have you really chew on issues — | am not sure what people walk away 
with beyond hearing about work on other campuses.” These comments indicated that AAU 
meetings focused more on local project activities, best practices, and implementation strate- 
gies, and less on delving deeply into professional debates about values and norms. In a few 
instances at AAU meetings we did observe groups based on academic discipline where partic 
ipants discussed their teaching. In these sessions participants sometimes got into heated philo- 
sophical discussions about the best ways to improve undergraduate STEM education. 


The few project sites that devoted more attention to deliberation about values and norms had 
been engaging in STEM reforms for many years. The AAU Initiative allowed these sites to con- 
tinue already existing conversations. These sites viewed the AAU Initiative as enhancing (but not 
originating) conversations about values and norms. 


Most project sites and campuses within the broader AAU STEM Network had not yet deeply 
explored existing values or the adoption of a new value system. While the AAU Initiative helped 
foster a new value system through new institutional logics, STEM reform projects, and dissem- 
ination via networking, the deliberation of changing values often necessary to ensure long-last- 
ing reform was seldom evident. The Initiative is still in its early stages and has time to make sure 
that space is provided in future meetings (as well as in feedback to project sites) to encourage 
deeper deliberation about cultural norms and values related to teaching. 


Networks. AAU successfully created and leveraged networks that shared best practices, pro- 
vided emotional support, enhanced resource development, shaped campus priorities, influenced 
leadership, and disseminated institutional log- 

ics. The networks created as part of the AAU successfully created and 


Initiative served many important functions leveraged networks that shared 
related to scaling change. fi ‘ : 

best practices, provided emotional 
The success of AAU with networking was man- support, enhanced resource develop- 


ifest in the develooment of new networks with «ae 
distinct groups including the AAU STEM reform ment, shaped campus priorities, 


community, the project sites, and the points of | influenced leadership, and 

contact. AAU also encouraged and assisted disseminated institutional logics. 
many of its member institutions in participat- 

ing in various networks that supported scaling STEM reform. For example, UC Davis interacted 
with BVA, CIRTL, The Helmsley Charitable Trust, the Keck Foundation and the Gates Foundation. 


Other interviews spoke about the value of AAU’s creation of a network of networks. AAU 
accomplished this by combining its networking and systems approaches. Rather than viewing 
networking in a limited way, AAU created and connected networks across the entire higher edu- 
cation system — from departments, to institutions, to the AAU network, to networks across the 
sector of higher education. One faculty member explained: 


“There were a couple of examples where I’ve seen this multiple network effect. For 
example, at the AAC&U STEM Education meeting I’ve seen groups from those eight sites 
presenting there and sort of vice versa where the participants of national organizations 
attend the AAU network meeting in Washington. The presenters are definitely a part of 
the larger community that I’ve seen either presenting or engaging in different ways in 
other networks. There's clearly this cross network happening. There's clearly—there’s so 
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many different networks around the country and AAU is intersecting with all of them. 
You just see these faces and influential people who are making changes. What really 
strikes me in those, | think, are the connections and the networking capacity that’s really 
kind of being stimulated by AAU.” 


In addition, AAU leveraged change through its relationship with the Coalition for Reform of 
Undergraduate STEM education (CRUSE), which is a rich resource for expanding and making 
available ideas for future reforms in STEM education. 


In addition to information sharing, networks played a valuable role in the AAU Initiative by help- 
ing project participants find emotional and political support as they pursue this complex task of 
changing undergraduate STEM education: “There is a sense of community that says we're not 
alone in trying to do this, that we're part of a bigger thing which is really nice. It does have the 
potential for sharing ideas back and forth, and we have done some of that.” 


External Supports/Intermediary Organization. AAU was partially successful in creating 
external supports. One of its key support mechanisms — the Framework — was very useful to 
project site Pls but not well understood by most other project participants. This result is not par- 
ticularly surprising because AAU is not typically engaged with the day-to-day work of the fac 
ulty, staff, and chairs of its member institutions. It is too soon to tell if other AAU support mech- 
anisms, such as the Essential Questions and Data Sources, will be well received by faculty and 
staff at member institutions. 


AAU's use of external support was often successful, including providing media coverage and 
public relations for the AAU campuses involved in reforming STEM education. Working with the 
Cottrell Scholars helped disseminate new ways of measuring excellence in teaching as well as 
stimulating discussions about the role of teaching in faculty rewards. 


Motivation and Sustainability. Theories of scale also speak to motivation through the devel- 
opment of incentives, ownership, and organic creation as well as sustainability through contex- 
tual supports and alteration of norms. Deliberation, networks, and external supports are the vehi- 
cles that typically create motivation and sustainability, the underlying mechanisms that support 
scale. In terms of motivation, the Initiative was perceived as organic in part because campuses 
provided input on the Framework, metrics to be collected, the RFP, and other core documents in 
the Initiative. As for ownership, evidence of which in this case we take to mean individuals feel- 
ing committed to engaging in or supporting innovative teaching in STEM, the Initiative achieved 
partial success. Ownership primarily was demonstrated by faculty champions and department 
chairs with less ownership by individual faculty and staff. Most campuses were unable to spread 
and obtain ownership by a large number of faculty. Signs of ownership by some senior leaders 
were evident at AAU meetings for presidents and provosts as well as in interviews with project 
Pls. Yet most individual faculty members and staff, including those significantly engaged in proj- 
ect work, were not aware of this commitment. One visible sign of institutional-level ownership 
was changes to physical spaces used for instruction. At some campuses, such as the University 
of Arizona, newly modified spaces have become prized locations for faculty members using active 
learning strategies. The most challenging task in modifying cultural norms at AAU institutions, 
not surprisingly, remains changing the faculty reward system, particularly by increasing the value 
of teaching. Although there were some signs of institutional ownership in changes made to fac 
ulty rewards, such as the requirement at some campuses to include additional materials about 
teaching in promotion and tenure dossiers, these changes were not widely noted by respondents 
and few of those commenting said that teaching had become more valued in promotion and 
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tenure decisions. Changing faculty rewards is undoubtedly the most important and contested 
aspect of long-lasting cultural reform on AAU campuses. Finding ways to institutionalize changes 
in incentive structures is a crucial part of the long-term success of the AAU Initiative. 


Sustainability of the Initiative is very much a work in progress. Respondents at many campuses 
reported initial signs of supports including professional development, technology, altered class- 
rooms, institutional commitment and other infrastructure. Successful completion of these sup- 
ports is an important factor in whether or not STEM reforms become institutionalized. Substantial 
evidence exists of support for teaching excellence, as evidenced by alternative metrics to mea- 
sure teaching and learning as well as improved standards for professional development. The 
underlying values that shape norms about the importance of undergraduate teaching are still a 
work in progress. Development and widespread acceptance of these teaching-related norms is 
fundamental to achieving the long-term goals of the AAU Initiative. AAU also demonstrated 
sustainability in several ways: the new AAU president continued support for the Initiative started 
by her predecessor, AAU changed the status of the director of the Initiative from soft money to 
hard money and added an additional 1 full-time equivalent of staff support, and AAU con- 
tinued to support some Initiative dissemination activities beyond the life of the original grant. 


Summary of Scale. Organizations working to scale reforms must assess their strengths and 
identify strategies that best fit their ability to encourage deliberation, networks, and external 
supports (if at all). This assessment must also identify activities for which the organization is ill- 
suited, and where possible identify alternative organizations or strategies to foster scaling. A 
good example of identifying a partner organization and negotiating its involvement with the 
reform effort is AAU’s work with the Cottrell Scholars. AAU supported the work of the Cottrell 
Scholars to develop measures of teaching excellence — a crucial part of reforming undergradu- 
ate STEM teaching — which was a task for which Cottrell Scholars were better suited than AAU. 
Forming collaborative relationships with other organizations better positioned than AAU to 
achieve some aspects of scaling STEM reform is potentially an effective way to promote delib- 
eration and implementation of STEM reforms on AAU campuses. 


Conclusion. The AAU STEM Initiative successfully employed two key mechanisms for scaling: 
networks and establishing a set of external supports. To date, AAU has been less successful in 
finding ways to enhance deliberation of cultural norms on member campuses. 


AAU's strength in scaling change is its unique position as a prestigious organization that can 
influence the field of higher education. In sum, although many of the models for scaling reviewed 
in this report have proven useful in other situations, AAU seems best positioned to use its influ- 
ence to scale changes in undergraduate STEM education. 


The findings from this study also suggest that trying to adopt all the scale strategies may not 
be a strong or meaningful approach. Instead, assessing an organization’s strengths and capa- 
bilities is the important first step to determining a scaling approach. If an organization is not 
well positioned to create deliberation, networks, or external supports, then forming partner- 
ships with other organizations to implement scaling strategies is important. Organizational asses- 
ments can help to identify approaches to change that best fit an organization’s strategic 
strengths and capabilities. 
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Maximizing AAU’s Role as a Change 
Agent to Achieve Cultural Change 


In this chapter, we review interviewees’ perceptions about the potential role(s) for AAU in encour- 
aging greater attention to teaching among its members. Respondents recommended that AAU 
first change its own culture to indicate its commitment to better teaching. The assumption by 
these respondents was that member institutions would fall in line if AAU as an organization made 
clear its preference for better teaching. These respondents also believed the clearest way to 
exhibit commitment to teaching was by making it a criterion for membership in AAU. 


Before proceeding to examine respondent recommendations, we note that most respondents were 
not well informed about AAU or its operations. To better understand the feasibility of their recom- 
mendations we start by clarifying how AAU operates. AAU is not an independent organization dic 
tating policies to its members. Rather it represents its members. General policies and funding are 
approved by a board consisting of presidents of AAU institutions. The AAU staff propose a bud- 
get and priorities; it is the board that authorizes them. If AAU were to expand its criteria for mem- 
bership to include metrics for teaching, the presidents of member institutions would have to initi- 
ate the process. Such a process may be possible; however, it is not a decision made solely by AAU 
staff. In addition, the metrics currently available to measure teaching quality are not widely agreed 
upon nor widely available, making it difficult to compare instructional quality at the institutional 
level. AAU can work (and in some ways has already) toward the development of such metrics. 
Without the necessary metrics it is difficult to make a proposal to the board to expand member- 
ship criteria. Finally, as described in the previous chapter, AAU made a real as well as symbolic com- 
mitment to undergraduate education by establishing the Initiative. This commitment included find- 
ing funding for the Initiative, convincing member presidents to support it, making the director of 
the Initiative a full-time staff member, and adding staff to assist her. Even though AAU had made 
some efforts that demonstrate work toward culture change, interviewees recommended further 
steps to institutionalize culture change in the long term. 


AAU’s Membership Criteria. The most often mentioned proposal for AAU to maximize its role 
in creating change was to expand its membership criteria, currently primarily based on research, 
to include teaching. One administrator described how the membership criteria could be a lever 
for change: “| think the meetings are good, the focus is good, but they need to really address the 
AAU membership criteria. It’s the elephant in the room. You can have this initiative and activities 
and conversations but if the membership criteria remain the same | just don’t see this having trac 
tion long-term.” A faculty member emphasized the importance of changing criteria for scalable 
change: “As | said before, | think in order for there to be sustained global change, the AAU has 
to have some metrics around teaching and learning that are meaningful that their institutions strive 
to achieve. | mean it is like the metrics they use for deciding who gets in. The research dollars and 
other sorts of things that they use to decide who is an AAU institution. If there is nothing there 
around teaching and learning, then | don’t think it is going to be sustained and | don’t think it is 
going to have as broad of an impact as it could.” Several interviewees spoke about the member- 
ship criteria as a barrier to change itself. For example: “Well, they can host meetings and write 
reports, but if they do not change the membership criteria, which say what they think is import- 
ant, nothing is really going to change. It will be lots of talk and some action and little impact.” In 
sum, the membership criteria were seen as both a barrier to change and a major lever in maximiz- 
ing AAU’s role as a change agent. 
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AAU’s Organizational Identity. AAU has evolved into an organization focused primarily on sup- 
porting research. AAU’s identity is heavily focused on supporting academic research at its mem- 
ber institutions. It has a long history of policymaking around research and advocating for research 
funding. Respondents found two aspects of AAU’s identity potential barriers to increasing the 
influence of the Initiative on undergraduate education. First, interviewees described the priority 
iven by AAU to research as a potential cultural . . 
ee to the improved undergraduate educa- Interviewees described the 
tion envisioned in the Initiative. As one inter- priority given by AAU to research 


viewee described: “| think they really need to 46 @ potential cultural barrier to 
reposition what the research university is. Right 


now, the organization defines it[self] as being the improved undergraduate 

excellent in research based on its criterionand education envisioned in the Initiative. 
priorities, but they could redefine that and say 

that a great research university is excellent in teaching and research. So, until they change their 

own definition of what a great research university is, | think efforts like this will struggle. So, if 

they want research universities to change, they need to change first.” We note that these respon- 

dents seem to consider AAU as an independent organization that drives the modern definition of 

the research university primarily through its control of membership criteria. 


The second major aspect centers on AAU’s role as a policy organization. In the past, this meant 
AAU focused primarily on policy issues and advocacy; little effort was spent on projects. The 
AAU Initiative was an initial step toward project management. Although at the beginning of the 
Initiative the former president, Hunter Rawlings, indicated that “AAU does not do projects,” 
over time the staffing and activities have expanded to allow more project-related work. 
According to some respondents, incorporating a project-based component to promote institu- 
tional change is an important step toward continuing the momentum developed by the Initiative. 
Others worry that this expanded focus might be detrimental to AAU: “I wonder if work with 
project sites is using their assets. It seems like they should focus on the existing infrastructure 
that they have —- what about getting a group of presidents to champion STEM reform, have a 
convening of AAU institutions to redefine excellence in university teaching, or get a network of 
provosts working on setting standards for excellence in teaching? They can work within what 
they're already doing and amplify it. Setting up some separate initiative also sets it up for just 
going away once the funds are over.” Interviewees questioned whether AAU was well-posi- 
tioned to run programs and work on institutional change efforts, and whether it would need 
to rethink its position on programs if it wants to continue this type of work. Recent structural 
changes at AAU, such as making the director of the Initiative a permanent position, adding staff 
to assist in project-related work, and seeking additional funds for the project indicate that AAU 
is taking steps to expand its capacity in this regard. 


AAU is well-positioned to use its influence because it has a history and tradition of working 
with advocacy. AAU’s work with presidents and provosts has focused on institutional priori- 
ties and networking. Work to shape institutional norms and leverage networks builds off AAU’s 
historic strengths. AAU has begun to add programmatic and project work to its policy and 
advocacy functions in part to play a more active leadership in higher education reforms. Going 
forward, the challenge for AAU is to determine if the addition of project-related activities comes 
at a cost to its historical ability to leverage influence or if the two are complementary. Especially 
important is judging the AAU’s capacity and resources relative to its various tasks. In the cases 
of the Initiative and a national survey on sexual assault on college campuses AAU decided it 
had both the resources and the appropriate stature in the academic community to do the work. 
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In contrast, in deciding whether to help develop a national departmental award for teaching 
excellence AAU decided it did not have the capacity to manage and fund such an award. 
Instead, AAU partnered with organizations that agreed to monitor and administer the depart- 
mental award. AAU took a similar approach to developing measures of teaching excellence 
with the Cottrell Scholars. 


One of the most frequently identified missed opportunities for scaling change was AAU’s lack 
of reflection on its own organizational identity. Interviewees asked AAU to consider how it his- 


torically has supported excellence in research . 
(actual programs and infrastructure at AAU) Interviewees asked AAU to 


and what the organization might do to pro- ~¢onsider how it historically has 


mote excellence in teaching. Rather than an fe iI 5 h 
externally-focused initiative, many com- SUPPOrted CZCerenoe In researG 


mented that changes are more likely to occur. (actual programs and infrastructure 


and be sustained if the Initiative was aimed at . . 
both cultural changes at AAU in what the at AAU) and what the organization 


organization values, recognizes and rewards Might do to promote excellence 
in conjunction with the external changes jn teaching. 

desired at AAU campuses. One interviewee 

summarized this point: “It’s my belief that AAU will eventually develop a ranking for teaching 
accomplishments. Once that’s established and can be measured, | think that’s the point in time 
where transformation will really take hold.” 


President and Provost Networks. Many respondents said that AAU could, in its regular 
convenings of member presidents and provosts, encourage them to give greater priority to 
teaching excellence on their campuses. Many individuals commented that communication with 
these groups about the Initiative was too infrequent and that they were not charged with any 
responsibility to ensure changes were happening on their campuses. Second, interviewees 
noted how a group of presidential champions might be tapped to push for an expansion of 
AAU membership criteria, development of a report about a 21st century research university, or 
other avenues that would signal changes within AAU values. 


Other interviewees recognized that finding ways to encourage senior leadership to add another 
task to their already full agenda was challenging. Some interviewees also recognized the dif- 
ficulty for AAU staff in trying to work with presidents and provosts. For example: “The pro- 
vosts really don’t need to see another talk about how all the stuff works [good teaching prac 
tices]. We know it works. We know what works and we're frustrated that we don’t have 
enough direct influence to just get everybody to change all at once. Doesn't really help to show 
the data again. And we've got all our own data [institutional based data]. It’s been measured 
so many times. We know that stereotype threat is real. We know lectures don't work. We know 
all these things are the right way to do it and | just worry that if the AAU- If we get to Laguna 
Beach and there’s another talk about this, | just think it would be, it might be, counterproduc 
tive.” Some respondents suggested that actionable work and examples might be more effec 
tive with senior leaders than conversations about valuing teaching. As a whole, interviewees 
believed that passively informing presidents and provosts of the AAU STEM Initiative is not as 
effective as actively engaging them in fostering change on their campuses. Perhaps AAU could 
develop a small group of presidential champions to reinforce the work done in the Initiative. A 
group of champions, for example, might support a competition to encourage AAU campuses 
to enhance their commitment to teaching. 
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Downsides of Exclusivity among AAU Universities. One challenge for the Initiative is to 
balance appreciation for context, taking into account institutional and departmental differences 
in implementation strategies, with the very real barrier resulting from this focus on exclusivity. In 
some cases, AAU institutions would not adopt successful reforms from other campuses because 
such reforms could not fit into the local context. Our data suggest that this attitude slowed 
change even when the external reforms might have been folded into institutional practice. We 
noted in previous chapters how models for teaching excellence from liberal arts colleges, as one 
example, sometimes were rejected out of hand because of the difference in institutional type." 


One key to enhancing the quality of undergraduate teaching at AAU institutions is to find a way 
to balance “respect for the research university identity and culture” and “unique campus con- 
text/ cultures” with being open to potentially transferable teaching practices from other types of 
institutions. Data analytics, for example, has proved transferable across institutional types. 
Professional development practices are also likely transferable across different contexts or insti- 
tutional types. Although more complex, even some aspects of teaching standards might be trans- 
ferable (in this example most likely across institutions of the same type). AAU could do more to 
emphasize transferable practices, policies, and ideas — even those that come from outside the 
sector — and the value of learning from examples such as University of Maryland, Baltimore 
County, which was an institution brought up repeatedly as an important example of excellent 
STEM education. As one respondent noted: 


“There's over 40 years of research and work on how to teach better. These institutions 
are going to make really slow progress if they feel they have to start all over again 
learning what we've already learned. Why can’t they learn from others? We know 
Uri Triesman’s collaborative learning model can work within any institution. Why 
aren't these institutions [AAU campuses] using all of these existing strong models 
rather than just reinventing them themselves.” 


The Initiative would likely advance and scale changes more quickly if it builds off of existing 
research on pedagogy and curriculum, borrows models from campuses such as UMBC, and legit- 
imates the expertise of centers for teaching and learning. Especially harmful is the impression 
that no reforms are valid unless developed at an AAU campus, which is constraining innovation 
and adoption of best practices. The elitism of AAU campuses can make them less prone to adop- 
tion, learning, and culture change from other campuses. 


Conclusion. This section highlighted key areas where AAU can increase its impact and poten- 
tially increase its ability to scale and sustain changes. The areas highlighted in this section build 
on the already-identified strengths of AAU — especially strategies that extend their influence and 
further tap key prestige groups. The recommendations amplify and build on existing networks 
of presidents and provosts, encouraging them to be more active agents of change or to form a 
network of peers among presidents and provosts to support improvement of teaching on their 
campuses. The recommendations also continue to expand AAU's important work in defining 
new institutional logics — challenging existing hierarchies that discount the value of teaching. 
The most commonly described and perhaps most meaningful in terms of altering the cultur- 
al-symbolic system is AAU altering its own cultural values in support of teaching. 


16 While interviewees did not as often directly describe attention to institutional context or culture as a barrier to culture change, 
researchers identified this trend in looking across comments of interviewees during questioning about adoption of new practices 
or learning from other sites or examples. We recognize that AAU was being attentive to institutional context in order to engage 
research universities by honoring that they have a distinctive environment which should be taken into account when thinking 
about STEM reform. 
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In this final chapter, we summarize and provide recommendations for other 
organizations attempting to scale practices, particularly initiatives related 
to improving the teaching and learning environment. For organizations 
engaged in scaling change, we have developed a practitioner guide with 
tools to help in this process. 


1. Assess your strengths and assets as well as limitations as an organization 

One of the most important lessons of this study is that organizations engaged in change pro- 
cesses should first assess their strengths and devise strategies accordingly. This lesson is espe- 
cially true when dealing with complex phenomena involving several organizations. Capitalizing 
on an organization’s strengths is crucial to effective change strategies. For AAU, these strengths 
included its ability: (a) to influence leaders such as presidents and provosts as well as other 
prestige and influence organizations; (b) to create and leverage networks; (c) to define over- 
arching logics or value systems for the Initiative; and (d) to work across the higher education 
system, ranging from the National Science Foundation, to partners in STEM reform such as 
the Bayview Alliance. 


The converse is also true: organizations need to identify their limitations and not take on 
strategies for which they maybe are less well-suited. For example, AAU was ill-suited to work 
directly with project sites because of its lack of resources and limited influence with local 
faculty and staff. Instead, site visits, which focused more on influence and information shar- 
ing, were a more effective way for AAU to work with individual project sites. 


2. Using a systems approach is promising to scaling change 

Every organization will have capacity to create change at some levels of the higher education 
system; few can work across the entire system. Some associations and organizations might be 
very well-prepared to work with individual faculty on professional development or curriculum 
reform efforts; other groups might be well-positioned to assist mid-level leaders such as deans 
and department chairs; other associations have regular workshops to help campuses institution- 
alize changes. AAU is one of the few organizations that can work across multiple parts of the 
higher education system and took advantage of that capability in its Initiative. Organizations 
should establish where they can work best within the overall system and strategically apply their 
efforts accordingly. 


Research has shown that the more levels of a system that are impacted, the more likely 
changes are to scale and be sustained. One of the beneficial outcomes of the AAU Initiative 
was better alignment among various reform efforts in undergraduate STEM education. 
Alignment of various parties involved in the same general policy arena (CRUSE is one example) 
— in this case undergraduate teaching in STEM — increases the odds of the scaling of the 
reforms. The efforts to align organizations begun under the AAU Initiative were promising. 
They will require ongoing resources to keep the alignment in place. 


Scaling Improvement in STEM Learning Environments: The Strategic Role of a National Organization 83 


3. Develop a multi-theory strategy for maximum impact 

Using multiple theories of action to scale change can be very valuable in projects with multiple 
stakeholders and complex motivations. AAU project leaders adopted a multi-theory approach 
to the change process, which supported the trajectory and efficacy of its reform strategies. 
Systems change, learning, influence, institutionalization, cultural change, and networks are 
complex approaches to change. It is all too common for change efforts to adopt a simplistic 
approach to change. AAU’s deployment of various theories of action increased the impact of 
the Initiative. As the AAU example shows, embedding strategies which can be used in multi- 
faceted ways is also an efficient way to use time and resources. Using multi-faceted strategies 
is a very effective way to incorporate multiple theories of action into a change process. 


4. Understand and intentionally plan influence strategies 

Few examples exist in the literature on change about the articulation, definition, and use of 
influence strategies. This study provided concrete examples of what an influence strategy 
might look like in higher education settings. The AAU influence-based strategy included 
setting up institutional competitions, peer and benchmark comparisons, branding, awards, 
site visits, and partnering with influential organizations, to name a few. Every organization 
can influence some set of groups. Identifying and targeting these groups is the most import- 
ant part of an effective influence strategy. As noted throughout this report, influence is 
generally an implicit strategy. Organizations typically do not conduct strategic planning for 
the use of influence even though it is an important lever for change. This report articulates 
influence strategies that organizations might consider. AAU is in a unique position to influence 
the higher education system; few if any other organizations in the higher education system 
have AAU’s influence. Yet any organization might benefit from planning an influence strategy 
by mapping individuals, groups, and organizations with which the organization has 
influence. 


5. Carefully evaluate the framing and language used to communicate the change 
Little research about change processes exists that relates to the importance of the language, 
framing, and messaging of the intended new value system. Although much has been written 
about the role of language in change at a single higher education institution, little has been writ- 
ten about the process of altering or rearticulating “institutional logics” to guide whole system 
change. Just as an individual institution must carefully craft a vision of change in its strategic 
planning process, scaling efforts must also articulate a “common agenda” and, even more impor- 
tantly, a compelling set of logics to enact that agenda. This study identified AAU’s process of 
articulating a new institutional logic and the ways in which it was both highly successful and less 
successful. The study found that attention to language is particularly important in scaling 
changes. Language embodies the new values, which become implicit assumptions and tacit col- 
lective norms which influence attitudes and behaviors. An important lesson from institutional 
theory is that the strongest and most sustainable way to scale change is to alter the underlying 
value system — termed the institutional logics. In sum, organizations attempting to scale change 
should carefully define their logics and craft appropriate language for enacting them. 


Differences in language can also result in barriers or issues that upend change processes. 
This study found that the language appropriate for implementing the Framework, as one 
example, varied by role group. These findings suggest the importance of working with dif- 
ferent groups involved in a change process to identify relevant language during early stages 
of adoption or even before adoption. Taking this step may reduce future misunderstandings 
that can adversely affect adoption and scaling. 
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6. Use networks to scale change 
Networks are important actors in the change process. They serve many functions including 
information sharing, dissemination of logics/new values, brainstorming, learning, emotional 
support, influence, safety in numbers for 
risk taking, and resource development. The The study also demonstrated 
study also demonstrated the benefit of the benefit of connecting multiple 


connecting multiple networks, providing one 
support for networks, and developing lead- networks, providing support 


ership to maintain networks. Being inten- | for networks, and developing 
tional about the use of networks in change leadership to maintain networks. 
strategies is crucial in the change process. 

Organizations often allow networks to develop organically rather than considering ways to 
make strategic use of them to scale change. AAU's effective use of networks in the Initiative 
came from careful planning of their various roles in the Initiative. 


7. Apply strategies to facilitate learning and adoption, and examine barriers to 

learning in change processes 

AAU used several strategies to encourage adoption through learning. The AAU found it dif- 
ficult for its Initiative to promote learning within its member institutions. Processes to facilitate 
learning at academic institutions may require the type of work with campuses that many 
organizations are not suited to conduct. For example, projects that successfully foster organi- 
zational learning typically use in-depth facilitated processes, set up inquiry teams, and build 
up organizational data and information infrastructures. In addition, the study identified how 
barriers can emerge to learning and adopting change. These barriers include: (a) the attention 
of the Initiative to the uniqueness of local context and culture made it easier for faculty and 
administrators to reject lessons for reform from other campuses; (b) competition among cam- 
puses sometimes made them leery to learn from each other; (c) distrusting research and data 
from other sources — again related to the perception of unique context; (d) AAU not being 
known as a leader in teaching and learning; and (e) difficulties in encouraging collective prob- 
lem-solving. Organizations can benefit from understanding the capabilities needed to help 
shape learning among the groups with which they are working. 


This study also suggests the potential benefit of identifying barriers to learning during imple- 
mentation and addressing them in a timely fashion. If some of these barriers had been iden- 
tified early on, AAU might have been able to change its messaging and approach. Creating 
feedback loops in the change process can help with organizational learning and the adoption 
of new practices. It can also help improve communication of logics, influence strategies, and 
optimal use of networks. 


8. Consider culture change at multiple levels 

This study found that changing AAU’s own culture — with greater attention paid to teaching 
excellence — was a potentially important and as yet not fully realized part of the reform pro- 
cess. AAU was more engaged in encouraging change within its member institutions than in 
its own reform, especially in the criteria used for its membership. AAU has made considerable 
investment in the Initiative and has restructured its staff and resource allocation to better 
align them with improved teaching. Getting its member presidents and provosts on board, 
especially in considering expansions to membership criteria, is a complex but necessary task 
both to spread the reforms supported by the Initiative and to reexamine the role of member- 
ship criteria in the overall change process. 
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The plenary speaker at one AAU STEM Network meeting, Stan Deetz, noted that “culture eats 
strategy for lunch.” He went on to say that STEM reform (within departments and among individual 
faculty) are more likely to occur if AAU (as an organization and its member institutions) changes 
too. He underscored that trying to improve undergraduate education without fundamentally 
changing the nature of the university — its culture — will always be an uphill battle especially at 
research universities. To support reform of AAU's culture the following relevant actions might be 
considered: expanding AAU's membership criteria, messaging and engaging president and provost 
networks, revisiting AAU policy priorities, and creating a position paper outlining the American 
research university in the 21st century. All of these proposed actions reinforce a more active role 
by AAU in fostering cultural change among its member institutions and as an organization. 


9. Create distributed leadership to improve STEM education 

The literature suggests that leadership throughout the higher education system is critical for 
scaling changes. Interviewees acknowledged the importance of leadership in scaling change 
at each level of the system: the faculty, academic department, institution, AAU presidents 
and provosts, and academic disciplines. Distributed leadership emphasizes the impact of 
leadership throughout the system in creating change. It does not distinguish between formal 
and informal leaders. Distributed leadership also includes external positions of authority 
including faculty at other institutions, professional association employees, staff at disciplinary 
societies, and those in formal positions of external authority such as disciplinary association 
leaders and policymakers. Facilitating the development of leaders at various levels is an 
important step of the reform process to improve undergraduate STEM education. The AAU 
contributed to this goal through its annual meetings for the AAU Initiative and for presidents 
and provosts. Additional funds and other resources likely are needed to extend leadership 
development across the relevant role groups in undergraduate STEM reform. 


Conclusion. The AAU STEM Initiative was a rare opportunity to examine the role of a national 
organization in improving undergraduate STEM instruction. This study provides important feed- 

back for AAU to hone its reform strategy. More generally we have identified lessons for other 
national organizations seeking to foster improvements in undergraduate teaching. These lessons 
potentially apply to regional or local organiza- 

tions as well. The key concepts for organiza The key concepts for organizations 


mapping strengths and capabilities, having a 


coherent strategy for influence, maximizing the are mapping strengths and 

use of networks, and using practices that max- capabilities, having a coherent 
aie aman: strategy for influence, maximizing 
The study also highlighted lessons for funders the use of networks, and using 
seeking to scale teaching reforms, including the practices that maximize learning. 
importance of funding a national group to 


align change efforts, and the need to establish or support an organization to coordinate leader- 
ship development at multiple levels of the higher education system. 


Some important lessons in reforming undergraduate STEM instruction apply only to AAU. AAU 
as a prestigious organization is uniquely positioned to influence institutions and garner media 
attention. AAU plays an important role in the entire higher education landscape. Few if any other 
organizations in the U.S. higher education system can shape institutional logics and encourage 
reforms of such complex topics as faculty reward structures. For these reasons, AAU is an essen- 
tial player in the process of achieving systemic change to undergraduate STEM education. 
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